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ABSTRACT 

After outlining the 1984 activities and results of 
the Center for the Study of Evaluation's (CSE's) Evaluation 
Productivity Project, this monograph presents three reports. The 
first, "The Administrator's Role in Evaluation Use," by James Burry, 
Marvin C. Alkin, and Joan A. Ruskus, describes the factors 
influencing an evaluation's use potential and outlines the steps the 
program administrator must take to increase the evaluation's use. The 
second paper, "Factors Common to High-Utilization Evaluations," by 
Ruskus and Alkin, identifies six of these factors: level of evaluator 
effort, evaluator's leadership behavior, user involvement, evaluation 
reporting, evaluator involvement in implementing recommendations, and 
user commitment to use. The final paper, "Reflections on Evaluation 
Costs: Direct and Indirect," by Alkin and Ruskus, describes how to 
identify and assign values to both direct and indirect cost 
components, and how costs should be considered in relation to 
benefits accrued from an evaluation. (BW) 
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ANNUAL REPORT: EVALUATION PRODUCTIVITY PROJECT, 1984 



This report outlines the results of the Evaluation Productivity i 
Project's 1984 scope of work and sets the context for- the deliverables 
we produced during this period. These deliverables reflect the following 
project activities: 

o Recommendations for Improving Evaluation Productivity 

o Organizing for Evaluation Use 

o Instances of High Evaluation Use 

o Costs of Evaluation 
In addition, the report describes progress on our two 1984 planning, 
activities which will lead to project deliverables in 1985: 

o Demonstration/Documentation of the Use Process 

o Preliminary Planning for Synthesis of the Knowledge on Evaluation 
Use 

Recommendations for Improving Evaluation Productivity 

As we reflect on our examination of the evaluation use process and 
the factors tnat influence use, we saw that the educational administrator's 
role 1n promoting use was extremely Important: In addition to 
context-organizational factors requiring administrator attention, many 
human and evaluation factors that affect the use process will require 
administrative attention and influence to promote an evaluation's 
program-level use. In the past, leadership 1n these last two factor 
domains had not traditionally been associated with administrative 
responslbll 1ty. 

We began to discuss this finding with other evaluation-use researchers 
and with researchers and practitioners 1n educational administration. The 
consensus was that our perception of the administrator's role in evaluation 



\ 



\ 




use was accurate and Important and carried Implications that administration 
researchers and practitioners need to understand and act upon 1n their 
work . 

Our next task was to begin informing the field a1*out how the 
administrator can contribute to evaluation use and to formulate tactics 
administrators can apply as they provide leadership 1n the use^ process. 
We Initially presented our thoughts on the Issue at the annual Meeting of 
the Evaluation Network/Evaluation Research Society 1n 1983. We maa^a more 
detailed presentation with concrete procedural suggestions to D1v1s1oh\A, 
Administration, at the annual meeting of the American Educational Resear 
Association 1n 1984. We produced a CSE Report on the topic. 

The positive reception our work received at meetings such as those 
mentioned above, as well as administrator Interest 1n understanding how to 
guide and promote an evaluation's use, and therefore Its productivity, 
prompted us to seek publication 1n a manner that would Impact the 
educational administrator. The outcome of this effort 1s that the paper we 
presented to AERA's Division A 1n 1984 — The Administrator's Role in 
Evaluation Use (CSE Report No. 225) — will appear 1n 1985 1n a special 
edition of Studies in Educational Evaluation devoted to "Evaluation as a 
Management Tool." This volume, edited by Dr. Naftall Glasman, Dean of the 
Graduate School of Education at the University of California at Santa 
Barbara, has the potential for Influencing administrators' perceptions of 
the purposes of evaluation and their role 1n Increasing its productivity. 

The paper cited above and its acceptance for journal publication 
constitute the deliverable for our 1984 scope of work component dealing 
with recommendations for Improving evaluation productivity. 
Organizing for Evaluation Use 

As we described in previous reports to the NIE, during 1984 we began 



to seek book -length, commercial publication of the Organizing for 
Evaluation Use handbook and workbook. The handbook summarizes the research 
on evaluation use, describes and exemplifies the factors Influencing use, 
provides a school -district scenario portraying these factors and how they 
might be Influenced to promote use, summarizes the procedures we devised 
for organizing an evaluation's use, and provides some examples and 
worksheets to aid the administrator taking responsibility for such 
organizing. 

The handbook, on the other hand, encapsules the results of our 
research and 1s primarily devoted to providing detailed examples and 
procedures, and a series of worksheets to be used during the evaluation 
organizing process. / 

Earlier In 1984 we began negotiations with Sage Publications to 
produce the Organizing for Evaluation Use handbook and workbook. These 
negotiations, Involving close cooperation between project staff and members 
of Sage's editorial and marketing units, were time consuming but extremely 
productive. They led to re-assessment of the market potential for a 
two-volume product, to reconsideration of the audiences for our work, and 
to numerous revisions and additions. 

For example, as Sage and CSE staff discussed the two-volume market 
potential, 1t became apparent that Sage's market Interest and CSE's concern 
for dissemination and application of findings might better be served by 
combining the two volumes — the handbook with Its extensive narrative and 
the workbook with Us amplified procedures — Into one volume. Therefore, 
our conceptual scheme for organizing an evaluation's use and the procedures 
and worksheets we developed to apply that framework are now merged Into one 
volume. 

Further, as CSE and Sage discussed the audiences for the published 



work, two principal considerations emerged. First, many of the factors 
d1 scovered to have a bearing on use exert that Influence regardless of the 
particular evaluation setting. Second, evaluation 1s Increasingly being 
applied 1n many different social and other program settings. We decided, 
therefore, to broaden the proposed publication's scope. 

To accomplish this end, productivity project staff developed a new 
scenario, running parallel with the original (which has been retained) but 
within a different context, to provide an additional bridge between our 
use-research findings and the procedures designed to promote evaluation 
use. 

The revised work, entitled A Handbook for Evaluation Decision Makers , 
has been accepted by Sage for publication 1n 1985. The book and the 
successful contract with Sage Publications constitute the deliverable for 
this facet of our 1984 work. 
Instances of High Evaluation Use 

As we outlined 1n earlier progress reports, we have analyzed the 
evaluation descriptions and accompanying documentation submitted to us 
as part of the High Utilization Evaluation Competition Initiated by CSE and 
Division H at the 1984 AERA annual meeting. We have produced a CSE Report 
which details the purposes and procedures of the competition, and which 
will stimulate future evaluator Involvement 1n the competition (see 3 
previous progress reports outlining the procedures for continuing AERA 
recognition for "high utilization evaluation"). 

In addition, the paper describes the results of our high utilization 
' evaluation analysis and profiles both the evaluations and the factors they 
share 1n common. The paper also discusses those factors appearing 1n the 
evaluation entries which have counterparts 1n our utilization framework, as 
well as possible additional Indicators of utilization. 

8 



The CSE Report (Mo. 240), entitled Factors Commo n to High Utilization 
Evaluations , represents the deliverable for this component of our 1984 

work. x 
Costs of Evaluation $ 

Earlier CSE work (see, for example, Alkln & Solmon, The Costs of 
Evaluation , 1983) has suggested some Issues Involved 1n attempting to apply 
existing cost methodologies to educational evaluation. We have noted, 
however, that given the Idiosyncratic nature of the evaluation use process, 
1t may be difficult to generalize cost implications, from one evaluation 

setting to another. 

We have produced a CSE Report which provides some further perceptions 
on the costs of evaluation, focusing on direct and Indirect cost 
Implications of various evaluation factors. This report (No. 239), 
entitled Reflections on Evaluation Costs: Di rect and Indirect, constitutes 
the deliverable for this component of our 1984 scope of work. 
Demonstration/Documentation of the Use Process 

As we proposed 1n our 1984 scope of work, the handbook cited above 1s 
being Implemented m^represe'ntatlve school districts throughout the country 
to demonstrate the use process. In each of the selected sites, a local , 
administrator has assumed the responsibility (1) for organizing a program's 
evaluation for use, and (2) for coordinating the procedures we are planning 
for demonstration/documentation at each local site. Site selection 

proceeded as follows: 

With the assistance of practitioners and researchers who nominated 
candidate school districts for participation in our demonstration 
activities, we compiled a nationwide sample of districts conducting a 
formal program evaluation. The sites were selected so as to provide 
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variety In features such as size, location, population, program emphasis, 
and evaluation needs. The sampling pool was sufficiently large to ensure 
that we would attain the eight to ten broadly representative sites that^we 
proposed. Next, wo conducted exploratory telephone conversations with 

district administrators (normally the superintendent) to discuss the 
project and possible district participation. Each administrator Indicating 
* Interest received a formal letter of Invitation and a description of the 
project Including respective CSE and district responsibilities 1n the 
demonstration-documentation effort (see Appendix A for copies of these 
materials). 

In each district accepting the Invitation to participate, the handbook 
was sent to a designated administrator who Is assuming responsibility for 
organizing the evaluation for use. These Individuals range from 
superintendents, to deputy superintendents, to directors of the program 
being evaluated. 

Through this. process, the handbook 1s being Implemented and the use 
process 1s undergoing demonstration 1n the target number of sites 1n 
different states covering the eastern, western, southern, and mid-western 
^regions of the country. 

Documentation of the use-demonstration process does not begin until 

< 

1985. However, CSE has begun to collect. the baseline Information necessary 
to that documentation effort. As described 1n our scope of work, the 
documentation efforts are geared toward describing the settings, factors of 
influence, use-enhancing strategies, and degrees of evaluation use 1n each 
setting. This Information, and other Information mentioned above, will be 
a part of the documentation deliverable provided to the NIE 1n 1985. 

U) 
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Preliminary Planning for the Synthesis of the Knowledge on Evaluation Use » 

As *e described 1n earlier progress reports to the NIE, our 
preliminary planning for the 1985 CSE-LRDC synthesis of the knowledge pn' 
evaluation use 1s underway. The synthesis 1s planned to take place <n a 
working meeting 1n which four or five evaluation-use papers win provide 
the nucleus for developing suggestions for Improving evaluation practice 
and use. 

Our preliminary planning to date has Involved staff of CSE, LRDC 
staff, and other synthesis contributors. So far our plans Include the 
following: 

Four or five major papers will be prepare^ so as to provide the 
nucleus described above. 

J)ne of the papers will be prepared by CSE staff (Alkln, Burry, & V v 
Ruskus). This paper will distill our findings on evaluation use over the 
past decade, present the conceptual model of use that we developed from the 
research base, and offer Implications for evaluatlo^ practice. s 

A second paper will be written by staff of LRDC (Cotfey 4 Bickel), • 
and will draw on that Center's technical assistance role 1n the Pittsburgh 
school's evaluation efforts. This ^per will Identify factors found to be 
relevant to use in the Pittsburgh setting and describe strategies adopted 
there to stimulate use. 

A third paper, to be prepared by Michael Patton, will synthesize the 
evaluation use research, in education and other settings, that has evolved 

over the past decade. 

A fourth paper, to be developed by Jean King, will focus on the 
Implications of research findings for improving evaluation practice. J 
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, We plan to share these papers among the presenters and other selected 

» 

experts who have contributed to evaluation use research, practice, or 
theory. This selected group, which wm also participate 1n the synthesis, 
wm Include such experts as Michael Kean, Nathan Kaplan, and Ross Connor. 
In addition, our NIE project monitor, Susan Klein, has agreed to be a 
synthesis participant. 

We plan to have the participants meet 1n California for a two-day 
working session 1n February of 1985. In this session, the contents of each 
paper, to be read in advance by all participants, win be discussed and 
weighed for breadth and accuracy.'"* ? 

Each paper win be summarized by the author (s) and critiqued by 
participants from the standpoint of their own research, knowledge, and 

« 

experience. This critique 1s seen as a vehicle for suggesting additional 
areas the synthesis may address, for conceiving Implications for the field, 
and for generating group-sanctioned suggestions for Improving practice. 

In this way, then, we win bring together 1n a working session about 
twelve evaluation use experts who win jointly contribute to the synthesis, 
Its implications, and recommendations for future practice. A court 
stenographer win record the entire proceedings. 

After the proceedings of the February meeting have been transcribed, 
we plan to initiate 1985 editorial work leading to a draft, book-length 
work 1n potentially publlshable form. The issues outlined above are among 
the topics to be fleshed out as we continue or joint synthesis planning 

with LRDC. V 
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i ummary 

Ouring 1984 we have provided a series of recommendatio ns for Improving 
evaluation productivity . These recommendations (CSE Report No. 225) will 
appear in a special edition of Studies in Edu cational Evaluation 1n 1985. 

Our work on or ganizing for evaluation use and the handbook guiding* 
such organization has been brought to a successful conclusion with the 
receipt of the contract with Sage Publications to produce our Handbook for 
Evaluation Decision Makers 1n 1985. 

We have documented Instances of high evaluation use 1n a CSE Report 
(No. 240) entitled Factors Common to High Ut ilization Evaluations. 

We have offered some additional thoughts on evaluation costs 1ti a CSE 
Report (No. 239) offering Reflections o n Evaluation Costs: Direct and 
Indirect . 

The pre-publication version of the handbook 1s being Implemented in 
school districts in different parts of the country as part of our efforts 
to demonstrate and document the use process . We are collecting baseline 
data 1n anticipation of our 1985 documentation needs. 

Our preliminary planning for th e knowledge synthesis working meeting 

1s well underway. 

During 1984, further, we have presented our findings at AERA and are 
continuing our dissemination activities to advance knowledge of the use 
process and to Improve evaluation practice. For example, we have proposed 
three presentations - two papers and a symposium for the 1985 annual 
meeting of the American Educational Research Association (see Appendix B). 
Our planned activities for 1985 offer additional prospects for formal 
publications. 
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Thank you for your Interest 1n participating 1n CSE's project on 
administrative organizing for evaluation use. As I mentioned on the phone, 
an of the research on evaluation use agrees that 1n order to build locally 
relevant uses Into a program's evaluation, a local administrator (e.g., a 
project director) needs to decide what these uses should be and organize 
the evaluation around them. 

I've enclosed a CSE report describing our research on evaluation use, 
the factors known to affect use, and the process of organizing for use. 
The report Introduces the product - Organizing for Eva |M«*1on Use; » 
Handbook for A dministrators - that your designated administrator-organizer 
will receive from us and follow as he or she works with the program 
evaluator to plan for and follow up on uses for the evaluation. 

I've also included an outline describing our project's overall 
structure and the role of the schools or districts working «^h us. CSE s 
role 1s not to influence how you evaluate your programs but rather to help 
ensure that these evaluations have a high use potential. To help with 
this, in addition to the Handbook, CSE staff will be available for tele- 
phone consultation, and I can make a short, orientation visit to partici- 
pating sites 1n the fall of 198;. 

We will ask participating schools and districts to help us document 
the use process. This will primarily consist of filling out an occasional 
(short) questionnaire and some telephone follow up, and asking 
administrator-organizer to maintain a record of correspondence, _ meetings, 
and so forth, between Mm/herself and the evaluator. This documentation 
will also provide some Information about .the project director and the 
evaluator, and describe the project being evaluated and its overall 
context. 



Please let me know as soon as possible 1f you and your district will 
work with us 1n this project, so I can send you the Handbook 1n time for 
your administrator-organizer to become familiar with its content before the 
school year begins. 

Again, thank you for your interest, I hcpe we can work together in 
this project and jointly offer an Important contribution to improved 
evaluation practice. 

Cordially, 



James Burry 

Senior Research Associate 



JB:klf 
End . 
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ORGANIZING FOR EVALUATION USE 

CSE has been conducting research on evaluation use for the past several 
years. This NIE-funded work has uncovered a variety of factors that affect the 
degree to which program evaluations and the Information they provide are used 
for local program needs. 

One of the central findings pinpoints the need for a local administrator 
(e.g., a superintendent; a program director) to share with the evaluator the 
responsibility for planning locally-Important uses for evaluation, and for 
organizing the evaluation in ways that help ensure that these planned uses have 
a strong chance of taking place. Without this local leadership, 1t is extremely 
difficult, and often impossible, to make local use happen. 

To meet. that adml ni strati ve-evaluat1ve need, CSE has developed a product 
called Organizing for Evaluation Use: A Handbook for Administrators. This 
handbook summarizes the relevant evaluation-use research findings; describes the 
factors found to have an effect or. evaluation use; exemplifies these factors at 
work in a variety of educational settings; and shows how an administrator might 
influence the factors to help promote evaluation uses. 

With these examples as a constant referent, the local administrator using 
the Handbook decides on the uses/users for the evaluation, decides which of the 
factors are likely to have an influence on use 1n his/Mr setting, and then 
follows a series of step-by-step procedures, with planning and worksheets for 
each, to make sure that these factors have a strong chance of working for the 
intended uses. These organizing procedures begin while the evaluation 1s being 
planned and continue throughout the program's evaluation. 

« 

NIE has asked CSE to Implement the Handbook in districts throughout the 
U.S.- to demonstrate this evaluation-use process. We are looking for districts 
which: 

• will be doing a program evaluation, 1984-1985; 

• will designate a local administrator who will be responsible for working 
with the evaluator to organize the evaluation for use around locally- 
relevant needs such as program planning and Improvement; curriculum 
evaluation; staff development; etc. 

We would like to select districts that vary 1n size and geographic loca- 
tion; we expect to select some sites whose evaluator Is a regular district/ 
school employee and others whose evaluator 1s hired externally; we are Inter- 
ested 1n having the Handbook used 1n both externally and locally-funded 
programs. 

For each school or district participating, CSE will supply the Handbook, 
will make staff available for telephone consultation, and arrange for a short 
visit to each district in the Fall of 1984. 

Each participating school or district will be asked to supply CSE with 
Information (via short questionnaires; telephone follow-up) to help us document 
the use process at each site. 

Project period is from July 1984 to August 1985. 

James Burry* Senior Research Associate 
(213) 206-1508 
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I mproving Evaluation Practice a nd Use Through Administrative Leadership 

"James Burry and Marvin c. ai Kin* 

Center for the Study of Evaluation, UCLA 



This paper takes the perspective that an evaluation's potential for 
serving program^ evel dedsl on-making needs requires administrator 
leadership and admlnlstrator-evaluator collaboration. That collaboration 
should guide specification of the evaluation's questions and the 
procedures selected to answer .them. The paper presents an organizing 
framework that administrators can apply as they plan and manage their 
program evaluations and evaluation uses. The framework describes and 
exemplifies factors which can affect the use-management process and 
demonstrates tactics that administrators can apply to moderate these 
factors 1n the Interest of use. 
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improvi ng Evaluation Practice and Use James Burry 

Th rough" A dministrative Leade rship Center for the Stuoy of 

JAmES BDRftV km mm.t. AU1H Evaluation 

UCLA Center for the Study of Evaluation 145 Moore Hall 

* University of California, 

Los Angeles 
. Los Angeles, CA 90024 

Objectives 

This paper examines the development of a framework for organizing evaluations 
to Increase their use at the local program level. The principal objectives of the 
paper are: (1) to describe a framework that administrators can use for organizing a 
program's evaluation to «meet specified uses 1n light of factors which can Influence 
these uses; (2) by reporting our observations on how the use -Influencing factors 
Identified 1n the framework are manifested and how administrators can deal with 
these factors to promote use; and (3) to demonstrate the use process to provide 
administrators with a means of managing their own program evaluations to meet 
specified uses. 

Perspectives 

A principal function of evaluation 1s to provide Information to serve^program 
decision-making needs. Administrative leadership and adm1n1strator-evaluator 
collaboration 1n specifying these decision needs and planning and conducting the 
evaluation around them Is vital to that function. 

A variety of factors — human, context, and procedural — exist in any 
evaluation setting and can Influence the degree to which an evaluation can success- 
fully be used to meet program needs. These factors Include evalffator role, admini- 
strator commitment to use, range of questions asked of the evaluation, appropriate- 
ness of the Information provided to program users. 

An evaluation has an increased potential for being put to use to the extent 
that a program administrator understands the full range of these factors and the 
broad kinds of Influences they can have; ascertains the specific effects they are 
likely to have In his or her own evaluation setting; .and applies organizing proce- 
dures which pay attention to these effects so as to enhance the degree to which tne 
evaluation and Its uses can be successfully managed. 

To assist the program administrator 1n this endeavor, the framework Presented 
and analyzed places use-Influencing factors Into patterns ^L^lSL'to liet 
conceptualization of the evaluation use process and which can be managed to meet 

specified program-level uses. 
Methods 

The framework and Its conceptual schema grew out of a decade of CSE research 
on evaluation use funded by the National Institute of Education. A series of eval- 
uation SS^tSStTcAlkln; Mil*, * White, 1979), an "•l«%?l« 1 **^ t,on 
study (Dalllak, 1980), and an evaluation-user survey (Stecher, Alkln, a nes J er ! 
1981) formed the conceptual underpinning of the evolving framework. These studies 



demonstrated the use process 1n actual evaluations, illuminated some of the factors 
representing various perspectives and degrees of influence on an evaluation's use 
potential, and uncovered the tactics adopted by evaluators and users to stimulate 
evaluation use. 

. It began to be apparent that some of the identified factors can potentially be 
moderated by an evaluator to stimulate the use process. However, it became equally 
clear,, across all studies, that a program-level administrator is frequently 1n a 
better strategic position tha/i the evaluator to stimulate evaluation use. But the 
extent 1 to which a program administrator can stimulate use will depend upon his or 
her ability to determine which factors will have a bearing in th6 particular eval- 
uation,; to plan specific evaluation uses/users for the evaluation in light of the 
factors! that will affect these uses, and to collaborate with the evaluator to 
organize the evaluation, its procedures, and its information reporting systems in 
ways* that will influence the factors in the direction of the intended uses. 

This conceptualization of the use process would clearly lead to high-utiliza- 
tion evaluations only if it could be supported by an organizing framework which 
recognized the constraints of school -level evaluations and the broader decision 
arena of educational administrators who would be asked to assume the use-organizing 
responsibility. 

To determine what such a framework might look like, we spent a year analyzing 
the literature on evaluation use (Burry, 1083). This analysis uncovered a broad 
range of factors, each of which had been identified in empirical studied and 
further substantiated 1n numbers of theoretical /conceptual works, which can 
influence evaluation use. I 

Our next task was to determine possible relationships among the identified 
factors artd to assess the extent to which individual factors or groups might be 
moderated in the interest of use. This Investigation, which examined various 
factors and their relationships 1n detail, allowed us to construct a set of core 
factors which are likely to have a bearing 1n most evaluations and which can be 
managed in the interest of use by a program administrator who assumes responsibi- 
lity for this task (Burry, 1983, 1984). 

Having specified the core factors of interest, we constructed a four-stage 
use-organizing framework for administrators which was validated and refined on the 
basis of reviews by evaluation practitioners and theoreticians as well as on the 
basis of try-out 1n actual evaluations. 

The research findings, organizing framework, core factors, and moderating 
tactics are now presented in a handbook for administrators for managing their 
program evaluations around a set of intended uses (Burry & Alkin, 1984; Alkin et 
al, 1984, 1n press). Final validation of the framework is currently taking place 
in a nation-wide sample of school- and district-level program evaluations in which 
the handbook is guiding the evaluation use process. 

Data Source and Results 

The studies described above, the careful review and try-out process, and the 
resultant framework all focus on the principal use-influencing factor previously 
described: The extent to which a program administrator assumes responsibility. 
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1n collaboration with the evaluator, for organizing the evaluation around intended 
uses Is one of the principal determinants of an evaluation's potential for use. < 
The administrator's organizing handbook demonstrates this conceptualization of the 
-tfse process. 

E ducational Significance 

Research already cited has. suggested that an evaluator working alone 
encounters many obstacles to use (Burry & Alkin, 1984). Research has also 
indicated that administrator leadership and admlnistrator-evaluator collaboration 
1s frequently required to realize an evaluation's potential for use at the local 
program level (Alkin et al, 1984). 

The conceptual schema described here and itsfoperatlonalizatlon in the 
administrator's handbook will guide that collaboration. The handbook demonstrates 
the use process 1n various settings, and contextual 1zes an organizing framework 
that administrators can apply to Influence evaluation practice and increase Its use 
potential 1n their own programs. 

Understanding the need for administrative leadership 1n the use process and 
demonstration of the forms such leadership can take will Increase the ability of 
educational administrators, and the staffs they manage, to monitor and Improve 
their Instructional programs via Improved evaluation practice. 
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Educational Researcher 



Precis- 



This paper utilized data generated from' the AERA Division H. 
competition, Excellency in Evaluation Utilization, in identifying 
factors significant to high utilization. The variables were 
compared to an empirically derived utilization framework. In 
addition to validating factors already identified in the 
fraro l5f ork ' several new indicators emerged which will be consider- 
ed for expanding the factor framework. The study has practical 
implications for educational evaluators in directing them to 
aspects of the evaluation process that have high potential for 
maximizing utilization. > 
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-Summary- 



Objectives 



The purpose of this study was to distill the variables 
common to evaluations that indicated success in generating high 
utilization. The* empirical basis of the analysis was the 
comparison of user validated high utilization evaluations with 
an empirically 'derived evaluation use framework. The unique 
feature of this study is that the variables identified emerge 
from a data base created by an AERA Division H competitioj^for 
high utilization evaluation. These data are documented by 
userr and evaluators as. critical to actual utilizations. 

» 

The study has both a pragmatic and theoretical focus; The 
findings will provide practitioners with empirically validated 
information for conducting educational evaluations in a way 
that will maximize utilization , and they will contribute to 
theory building in evaluation utilization. 

Perspective 

v , - . ■ r- 

The definition of use employed in this paper is one 
developed by Alkin (1975) and Patton et al. (X978) . Use is not 
limited to the direct application of evaluation findings to 
immediate decision making, but rather it includes the supporting 
role often played by evaluation findings in the multivariate, 
incremental process of decision making. ... This perspective on 
utilization was further developed by the Center for the. Study of 
Evaluation through, a series of evaluation case studies (Alkin, 
Daillak, & White, 1979) , an evaluator field study (Daillak, 
1980) , and an administrator user survey (Stecher, Alkin, & 
Flesher, 1981). Based on these empirical data, Alkin et al. 
(1983) identified and classified the factors affecting evalua- 
tion use into three interrelated categories — human (evaluator 
and user) , context (pre-existing evaluation bounds, organiza- 
tional features, and project characteristics) , and evaluation 
(procedures, information dialogue, substance of information, and 
reporting) . 

The data provided by the evaluators of the studies reviewed 
for this analysis were analyzed in the context of this factor 
framework. While a largfe number of the utilization factors were 
validated, several new indicators emerged which will be consid- 
ered further for incorporation into the rnqre comprehensive factor 




framework • 



Data Source 

Twelve evaluation studies were used as the basis of this 
analysis. Summaries of these studies and supporting materials 
from relevant users were submitted to the AERA Division H Award 
for Excellence in Evaluation Utilization competition for 1983. 
While all of the evaluations dealt with educational programs, 
the majority (seven) • focused on public school programs at the 
district level-. One addressed a state-wide educational program, 
three were directed at university programs, and one targeted 
vocational education for a special needs population. Three 
of the studies specifically focused on teacher effectiveness. 

Application for the competition required each evaluator 
to discuss unique factors which they judged to be significant 
in contributing to the ultimate utilization of their evaluation. 
These data were » ulyzed for the purpose of this study. 

Method 

Applicants' narrative data on unique factors contributing 
to utilization were analyzed by abstracting all statements made 
in the application and documentation. Data were given equal 
weight regardless of the applicant's placement in the utiliza- 
tion competition since all applications represented well 
documented instances of utilization. The indicators represent- 
ed in the data matrix were then classified according to the 
utilization framework. Most indicators ware easily subsumed 
under existing major factor categories, although some indica- 
tors presented extensions of existing factors or their 
interpretation. -Frequencies were- then tabulated by. factor. 

Results 

The results confirmed many of the factors identified by 
Alkin et al. as significant. The citing of factors within the 
evaluator' s control as most significant constituted one dominant 
theme in the findings. Evaluator characteristics accounted for 
31% of use and evaluation factors accounted for 52% of use. 
Factors outside of the evaluator' s direct control (user charac- 
teristics and contextual factors) accounted for only 17% of use. 
However, we^should keep in mind that in a self report schema it 
is likely that personal rather than the contextual characteris- 
tics would predominate. 

Among evaluator characteristics, willingness of the evalua- 
tor to involve users in the evaluation was the most frequently 
cited attribute. Two facets of the evaluation stood out as 
critical to use: evaluation procedures and evaluation reporting. 
The most frequently highlighted evaluation procedures were the 
evaluation plan, evaluation methods, and follow-up activities. 
The most frequently identified reporting features were the 



frequency of reports, the format of reports, the readability 
of report narrative, and the dissemination of reports. 

Several indicators were identified in this analysis which 
were either not included in the factor model or which expanded 
the meaning of existing factors. These indicators are noted 
along with the existing related factor in parentheses. They 
are: sense of urgency or need for the evaluation to improve an 
ailing program (external organizational feature) ; a structured 
evaluation plan (evaluation procedures) ; dissemination of the 
report (evaluation reporting) ; and follow-up activities after 
the evaluation is completed (evaluation reporting) . 

Educational Significance 

Increasing the extent of utilization of educational program 
evaluations has clear implications for improving programs. This 
empirical validation of utilization factors has practical impli- 
cations for practitioners. Evaluators should focus on assuring 
user's working involvement in the evaluation, designing an 
evaluation plan that accommodates the information needs of all 
users, utilizing evaluation methods that are technically valid 
but congruent with program practices, generating readable 
reports, and conducting follow-up activities. " 

The study has implications for evaluation utilization 
theory as well. First, the study validated the existing utiliza- 
tion framework and suggested modifications and needed 
amplification. The findings also highlighted a possible 
evaluator blind spot — contextual variables. Exploration of 
factors outside of tt\e evaluator* s direct control and ways these 
factors can be successfully influenced to enhance use is an 
important step in promoting more widespread utilization of 
evaluations. 
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This symposium synthesizes and contextual 1zes the knowledge of the 
evaluation use process which has evolved over the last decade. Each of the 
participants has been active 1n conducting research on evaluation use 
and/or conducting h1gh-ut1l1zat1on evaluations. Presentations offer a 
framework for organizing, evaluations to promote their use; discuss examples 
of use from actual evaluations, Including the factors Influencing use and 
strategies for moderating them to stimulate use; provide a state-of-the-art 
synthesis of the principal use findings 1n education and in other disci- 
plines; and discuss implications for the practice and uses of evaluations. 
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A Synthesis of the Knowledge on Evaluation Use 

/ 

The purpose of this symposium Is to present a variety of current 
Information reflecting knowledge of the evaluation use process and ways to 
Improve evaluation practice and Increase/ Its use. 

The principal objectives of the symposium are as follows; 

° To present a framework for organizing evaluations for use in light 
of various influences on the juse process; 

• To provide examples, from actual evaluations, of the use process, 
as well as factors Influencing use and strategies for moderating 
these Influences to promote/ use; 

• To provide a state-of-the/art synthesis of the principal findings 
on use, in education and in other disciplines; 

• To discuss implications /for the practice and uses of evaluation. 

The symposium chair, presenters, and discussants bring a wide and 
varied background to the stuoV and practice of evaluation. The UCLA Center 
for the Study of Evaluation presenters, Alkln, Burry, and Ruskus, have been 
conducting research on evaluation use for the past decade. The Pittsburgh 
Learning Research and Development Center presenters, Cooley and Bickel, 
have a long and successful /history of conducting evaluations with high ' 
levels of use. Each of these two organizations has conducted their work 
under NIE auspices. Patton is one of the earliest evaluation use 
researchers whose work continues to Influence the conduct of evaluations. 
The chair and discussants have published In the field of evaluation use. 

The symposium summarizes, contextual izes, and updates a large and 
growing body of research on evaluation use. Another Important contribution 
of the symposium, however, Is the particular evaluation perspective It 
embodies. That Is, the symposium demonstrates how evaluations should be 
conducted to meet particular uses in various settings. This demonstration 
has Implications for both the providers and the users of evaluation 
Information. 

The three major presentations are as follows: 

Improvin g Evaluat ion Use: Implicati ons for Practice 

Warvin C. Alkln, ProjecT 01 rector fJames Burry, Senior Research Associate; 

Joan Ruskus, Senior Research Associate 

UCLA Center for the Study of Evaluation 

CSE staff have been conducting research on evaluation use 1n a. series 
of empirical Investigations over the past decade. For example, on the 
basis of evaluation case studies, evaluator field-observation studies, and 
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evaluation-user surveys, staff Identified many factors which can act f 
Independently or 1n concert to have a marked effect on an evaluation s 
potential for useL These factors, consisting of human, context, and 
evaluation procedural considerations, require careful attention In order 
for an evaluation to be used. 

How that attention might be applied has been the subject of recent CSE 
investigation. That Investigation led to the development of a framework 
which can be used to organize an evaluation to meet specified uses given 
the factors likely to affect these uses 1n the particular setting. 

This paper summarizes the CSE research leading to the 6rgan1z1ng 
framework. In particular, It describes and exemplifies a set of core 
factors which, since they are likely to have a bearing In most evaluation 
settings, form the basis of the organizing framework. The^paper describes 
how the organizing framework Is used, offers procedures for Improving 
evaluation practice and Increasing its use, and discusses Implications for 
the field. 

Evaluation Use: Pittsburgh Case Histories 
wTlTfaVBooley and William Bickei, senior Scientists 
Pittsburgh Learning Research and Development Center 

For the past several years, members of the evaluation unit at LRDC 
have been Investigating strategies for Improving the content and use of 
evaluation research in school districts. This goal has been pursued 
through collaboration with the Pittsburgh school system. The evaluation 
unit has taken on a number of Pittsburgh research tasks, ranging from 
evaluations of program Impact to- district-wide and program-focused needs 
assessments, designed to help managers establish district and program 
priorities. What has been learned about district use of evaluation 
research Is being summarized In a series of case histories. 

This paper draws upon several of the Pittsburgh case histories to 
discuss the issue of evaluation use. Specific examples of use are provided 
from Individual cases. Factors that seemed to play a critical role in the 
use of research Information are Identified. One example of such a factor 
concerns the importance of taking a client orientation 1n the organization 
of evaluation research. The paper places emphasis on the discussion of 
research strategies that can increase the likelihood of use, and of actions 
that both researchers and managers can take to Improve the utility of 
evaluation processes for school systems. 4 

State-of-the-Art in Evaluation Use 
MTcWerQTTa'tton, Program Director, CAEP 
University of Minnesota 

This paper summarizes the principal findings on evaluation use that 
have evolved over the past decade. It complements the other symposium 
papers by reviewing the research on evaluation use 1n education, and 
supplements them by drawing from the research on use conducted within other 
disciplines. The author, as one of the early pioneers 1n the field, is in 
an Ideal position to comment on and synthesize the current state of 
evaluation utilization research. 0 
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The paper covers the entire decade of research on evaluation use, 
beginning with some of the earlier research which the author himself 
conducted. Among this work was his 1975 analysis of evaluation utilization 
1n health evaluation* and his 1978 Utilization Focused Evaluation which 
described the early research and summarized the then extant literature on 
use. From that vantage point, this paper provides the opportunity to 
update the evaluation use literature, both 1n educational settings and In 
other disciplines, and to suggest the Implications of the findings. 



The symposium chair, Alex Law, Is head of the Office of Program 
Evaluation and Research, California State Department of Education. 

i ' 

The discussants are Michael Kean, Publisher, CTB/McGraw-Hill , and 
Susan Klein, Senior Associate, National Institute of Education. 

The chair ajd discussants themselves bring another level of expertise 
to the topic of evaluation use, as represented 1n publications reflecting 
problems 1n district-level uses of evaluation; the administrative uses of 
evaluation; and federal involvement 1n evaluation use. 
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INTRODUCTION 

The theme of this journal Issue — evaluation as a management tool 1n 
education — 1s both critical and timely. It Is critical because, "as the 
shift continues from the federal to the state levels in the management of 
education programs, the states become more, not less accountable for them. 
SEAs and LEAs have become accustomed to the federal government not only 
requiring the evaluation of programs but also dictating methods of evalua- 
tion" (Council of Chief State School Officers, 1982). As federal control 
of evaluation diminishes, and as federal resources for these evaluations 
also diminish, 1t seems likely that state and local administrators will 
need to assume greater responsibility for their evaluations (Burry, 1984). 
Given scarce resources, they will need to think strongly about the best 
ways to commit people and money to ensure thajt their evaluations generate 
useful information. One primary Index of that usefulness will derive from- 
the extent to which evaluation becomes a tool for educational management 
and decision making. 



'the research guiding this article was supported under a grant to the 
UCLA Center for the Study of Evaluation from the National Institute of 
Education, U.S. Department of Education. However, the findings and 
opinions expressed here do not necessarily reflect the position or policy 
of the National Institute of Education, and no official endorsement should 
be Inferred. 
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The critical 1ty of the Issue 1s also matched by Its timeliness. That 
1s, to suggest Increased administrator responsibility 1n evaluation without 
offering means to channel that responsibility would be less than satisfac- 
tory. Fortunately, recent research 1n evaluation use (Alkln et al, 1985 1n 
press) has enabled us to develop a framework which helps administrators to 
take a more active, Indeed proactive role, 1n organizing eva^atlons to 
Increase their effectiveness as a management tool 1n educational decision 



Recent attention to the quality of education in our schools (Boyer, 
1983: Goodlad, 1983; National Commission on Excellence 1n Education, 1983) 
indicates that concern with excellence 1s pervasive. That pervasiveness"* 
suggests that educational decision makers will need Information which 
accurately reflects the extent to which their educational systems are 
responding to the challenge for excellence. Can evaluation become a useful 
tool in the management of these systems? We believe that 1t can, if 
educational administrators take the kind of actions we discuss here to 
capitalize on the uses to which evaluation can be put and to ensure that 
these uses reflect their own system-level questions and needs. 

Establishing a local focus for evaluation presents little 1n the way 
of technical difficulties but may run counter to present attitudes toward 
and expectations for evaluation. We will elaborate some of the relevant 
issues later 1n our discussion. Suffice it to say, for the present, that 
1f evaluation is to address the demonstration of excellence (or any other 
desirable quality) then there should be open discussion of issues such as 
the definition of excellence, how it is to be judged, the best means of 



making. 
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demonstrating Its attainment, and the uses of evaluation 1n addressing t 
these Issues. 

4 

Bursteln (1984) has recently discussed some applications of evaluation 
1n school improvement efforts. While these applications can play a 
potentially central role 1n the definition, judgment, and demonstration of 
excellence, that potential will not be realized urrtll administrators come 
to recognize the ways 1n which evaluation Information can be used for 
management purposes. Among these .purposes are: pulse monitoring 
treating evaluation Information as educational Indicators of the extent to 
which the educational system 1s moving 1n the desired direction; student 
decision making — using evaluation Information to make accurate decisions 
about student progress and. needs; program decision making drawing on 

» 

evaluation Information to monitor programs and services and to modify them 
as needed; Informing educational policy -- using evaluation Information to 
guide discussion of the status of educational systems and mechanisms for 
Improvement; and long-range planning — applying evaluation Information 1n 
decision areas such as physical plant needs, teacher hiring and assignment, 
and resource allocation (Bursteln, 1984, pp. 16-19). 

As Bursteln suggests, however, such management applications, which 
address Instructional Issues, support systems, and resource allocation at 
both policy and operational levels, face certain requirements. Among these 
requirements are commitment to the uses of evaluation Information for 
"informed Inquiry and educational change," and "a healthy and Informed 
understanding of the limits as well as the possibilities of Information- 
based decision making" (Bursteln, 1984, p. 23). We will amplify these and 
other requirements as we proceed. 
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Enabling educational administrators to organize their evaluations to 
meet the uses outlined above, then, 1s the tpplc of our discussion. Recent 
research (Alkln et al, 1985 1n press) has uncovered factors which Influence 
the extent to which evaluation 1s likely to be put to use. One finding, as 
we shall see, 1s that the educational administrator, him- or herself, is a 
critical factor 1n the use process. That 1s, the extent to which the 
administrator actively Influences the direction and course of the evalua- 
tion 1s a principal determinant of the likelihood that 1t will be put to 
use. Other research (Ruskus & Alkln, 1984) suggests the kind of admlnls- 
tratlve Influence likely to promote system-wide use. Recognizing and 
promoting the uses to which evaluation can be put 1s the first step in 
establishing Its application as a management tool. 

Once accepted as a management tool, evaluation can provide a valuable 
resource for administrators who are Interested 1n finding out how well the 
system they are responsible for 1s running, and deciding whether 1t could 
be Improved. We have suggested that these decision needs can Involve, for 
example, monitoring student and program decision making, policy setting, 
and long-range planning. To meet these kinds of needs, an evaluation 
should be planned around questions reflecting the system's context, 
operations, and expectations. It must be conducted 1n ways to ensure, 
first, that these questions are answered and, second, that the answers can 
actually be put to use in making decisions about whether the system should 
continue to run as is, 1f 1t needs to be modified, what these modifications 
might entail, and what kinds of policies and resources might be required 
either for maintenance or modification. 



Although there are potential obstacles facing the administrator who 
wants an evaluation emphasizing sucha practical uses, there are also 
organizing principles that can be applied to overcome. these obstacles. As 
we discuss the administrator's role 1n organizing an evaluation we will 
suggest ways to strengthen Its potential for use. For purposes of discus- 
sion, we focus our remarks primarily at the level of a discrete education 
program such as, for example, Chapter I, bilingual education, mathematics, 
language arts. 

Organizing for Evaluation Use 

To have a high potential for use, an evaluation needs to be carefully 
planned, organized, conducted, and communicated to likely users of the 
information 1t provides. This kind of evaluation rarely happens by chance; 
someone has to take the responsibility to make 1t happen. Certainly, an 
evaluator can and should take some of the responsibility for organizing an 
evaluation for use. However, it has become clear that the role an adminis- 
trator (e.g., a superintendent; a Chapter I program director) takes with 
regard to the evaluation has a marked effect on Its use potential. We will 
demonstrate a framework, then, that administrators, working 1n cooperation 
with their evaluators, can apply to gain tactical Influence over the 
direction the evaluation takes. That Influence 1s Intended to Increase the 
evaluation's potential for use in program management and decision making. 
Evaluation Purpose 

The framework we propose reflects a particular evaluation perspec- 
tive. That 1s, we define evaluation as a means of providing information 
that can be used to make decisions about programs. These decisions might 
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stem from questions about whether the program could be Improved, they might 
reflect matters of resource allocation and monitoring, they might stem from 
questions about whether the kinds of attitudes people have about the 
program could be Improved. 

Using evaluation Information to assess and perhaps Influence 
participant attitude 1s likely to make an Important contribution to 
evaluation's potential as a management tool. Some programmatic changes 
will require modification of participants' attitudes about the program or 
Its evaluation before the change can be Implemented successfully. That is, 
an information-based administrative decision to make some currlcular 
change, to redirect resources, to reassign staff, will require staff 
support of the proposed change. 

Tha'w support will require acceptance of the Information driving the 
change, and frequently use of that information by the people who, in 
addition to the administrator, will play some role 1n accepting and 
implementing the change. Therefore, administrator ability to successfully 
use evaluation as a management tool will require colleglal support and 
information use at various system levels. 

For an evaluation to meet Its potential as a management tool in a 
particular decision area, then, the administrator needs to identify other 
potential information users whose support of the decision is necessary. 
For each user or user group, the administrator will need to determine what 
questions and concerns they have with respect to the given decision area, 
and then make sure that the evaluation applies procedures and reporting 
techniques which are appropriate to the users and their questions. 
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The extent to which a program administrator takes responsibility for 
identifying the intended users, determining their questions about the 
program, shaping the evaluation procedures for answering the questions, 
deciding what kinds of Information win be collected, and ensuring that the 
Information 1s effectively communicated can profoundly affect the degree to 
which the Information can successfully be put to use. 
Evaluation Use 

By use we mean applying evaluation Information to the resolution of 
the kinds of problems, questions, or concerns we have alluded to above. 
To be sure, evaluation can have other, perhaps unintended consequences, but 
we do not emphasize them 1n this paper. 

There arejnany potential users of evaluation information. In a school 
setting, for example, there might be a variety of programs in operation, 
such as: a Chapter I program; a state-funded bilingual program; a remedial 
math or language arts program designed for students 1n need of specialized 
instruction. 

Each of these programs might be evaluated and each could have a vari- 
ety of potential evaluation users. For example, let's assume that a 
district superintendent wanted to have more productive evaluations, wanted 
to be able to use evaluation Information as a management tool in district 
operations. The administrator might then consider ways to organize the 
evaluation to meet his or her questions and needs and those of other 
potential users. These users, 1n addition to the superintendent, might 
include the people responsible for program operation, for Instance, such as 
the director, other administrators, curriculum developers, Instructional 
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staff, and funding agencies. Other users might consist of parents, 
advisory councils, and community organizations wUh an Interest 1n the 
program. Since each of these groups can have professional and personal 
Interests 1n the program and Its evaluation, each is a potential user of 
the Information 1t provides. A>central concern 1n organizing for 
evaluation use, therefore, 1s the selection of the Intended users of the 
evaluation. 

Evaluation Information can be used 1n a variety of ways. For example, 
let's follow the case suggested above and assume that the superintendent 1s 
concerned about the Instructional content and methods used 1n a remedial 
mathematics program. As a responsible manager, the superintendent has 
questions about how students are selected for the program, the extent to 
which teachers are Implementing the program as planned, the extent to which 
building principals support teachers as they attempt to Implement the 
program, the extent to which resources earmarked for theVogram are 
actually used 1n the program, whether or not the program seems to be 
beneficial for the students. The superintendent wants the program's 
evaluation to provide answers to these questions so that he or she can make 
Information-based decisions about maintaining the program as 1s, modifying 
the program, maintaining, Increasing, or reducing Its level of resources. 

Now the evaluation may ultimately find that the program seems to be 
running quite well. On the other hand, 1t may pinpoint problems and 
suggest areas for change. Regardless of the evaluation findings, staff 1n 
the program are likely to have different conceptions about the program. 
Some may enjoy working 1n the program, think It's a good one, and would 



T1ke to see 1t being continued; other staff may take the opposite point of 
view. S'taff are also likely to differ In their expectations for the 
evaluation. Some way want information to help them as they carry out their 
responsibilities In the-p^gram; othervmay think, that evaluation does not 
provide the kinds of Information they need; some may have no expectations 

for the evaluation. 

In short, program staff, as potential Implemented of the 
superintendent's decisions, are potential evaluation Information users. 
They can differ In the extent to which they have questions about the 
program, 1n the kinds of questions they have, and their disposition toward 
using evaluative answers. For some staff, asking them to make changes may 
create a problem for the superintendent; for others, asking them to 

continue current practice may create a problem. ^ 

To help preclude these possibilities and to promote the kind of 
support we mentioned earlier, Involving staff and. other potential users 1n 
the evaluation, finding out their questions and concerns, and determining 
the kinds of information they are likely to accept and use, are crucial. 
F Affecting Use 

in any setting, there are many factors that can have an effect on 
evaluation use and therefore on Its potential as a management tool. By 
factor, we have 1n mind any characteristic or element present 1n a given 
situation that can affect the extent to which the evaluation is used. 
These factors stem not only from the conduct of the evaluation, but also 
from the surrounding social, political, organizational, administrative, and 
programmatic context. Factors potentially affecting an evaluation's use, 
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for example, Include the kind of role the evaluator chooses, the intended 
users' views about the program being evaluated, the various requirements 
for %he evaluation, and Its proposed methods. 

If these factors are accepted as glvens, they can reduce or negate the 
evaluation's use potential. For Instance, 1f an Intended group of users 
firmly believes that a program could not possibly be Improved upon, 1t may 
be difficult to convince them to modify their view, no matter what the 
evaluation findings might reveal. On the other hand, 1f the evaluation 1s 
structured and organized around Intended users and kinds of uses, and 1f 
the possible effects of various factors on the use potential are planned 
for, then the evaluation's likelihood for use can be greatly Increased. 

Later we will describe the full range of factors that have been shown 
to affect an evaluation's use and discuss an organizing framework 
administrators can follow to*1n1m1ze negative factor influence and 
strengthen positive factor Influence. The organizing framework, as well as 
the associated operating terms we have discussed above, grew out of our 
research on use over the past several years (Alkln et al, 1985 1n press; 
Burry, 1983). 

BACKGROUND ON EVALUATION USE 
For a good number of years, the terms use or utilization have been 
cropping up 1n the evaluation literature. Up to about the mid 1970's, 
however, discussions of use relied fairly heavily on Impressionistic and 
anecdotal information. Tfcere was a lot of talk reflecting what people 
thought use looked like, with explanations often relying on speculation 
(Rossi, 1972; Mann, 1972; Cohen & Garet, 1975). 
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Around the mid 1970' s the picture began to change. Then we began to , 
see the results of 'systematic, research on use, research trying to discover 
what use actually means, whether or not 1t occurs, and what works for 1t or 
against 1t (Alkin, 1975; Patton et al, 1975). 

To a great extent, the careful study of use grew out of the kinds of 
promises made for evaluation. Fof example, evaluation was to be an Impor- 
tant tool for decision making and for Improving policy and practice. All 
the evaluator had to do, it, was thought, was to provide valid data. People 
would see the light and use the Information provided; decision making would 
be more rational and policy and practice would improve. 



By now we know this was a naive 7 view. Certainly, Information vali- 
dity, especially when that term is mutually agreed upon by evaluator and 

» 

potential user, can contribute to use. But so long as evaluation and Its 
use were (1) seen as the sole responsibility of the evaluator, and (2) 



One of the things explaining the seeming lack of use was that for a 
long time many people thought that information received was necessarily put 
to use, and put to use quickly. When that did not bear up in practice, it 
was assumed that no use was taking place. 

As the research was to show, however, use was occurring, though 1n a 
form quite different from and perhaps more modest than had been expected 
(A1k1n et al , 1974; Patton et al, 1975). We began to understand that 
evaluation processes and evaluation information usually accumulate over 
time before they are finally put to use. And even when they are used 1n 




expected to produce quick, observable, |nd rational decisions in action, 



the promise was not met. 
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making a decision, that decision may also have- been Influenced by other 
kinds of Information and forces outside of the evaluation. This kind of 
use can and does take place and when It does 1t can help to Improve 
educational decision making and practice. 

However, there 1s something else that helps explain lack of use. That 
1s, for use to take place, we had thought, such technical factors as the 
quality of the evaluation's procedures would be important. And that 1s 
true. Procedural soundness can certainly contribute to use, but so can 
other factors, factors that are somewhat removed from the technical realm. 

For example, one early CSE finding (Alkln, 1975) showed that the 
stance taken by the evaluator with respect to a program's social context 
can affect the evaluation's use potential. Concurrent research (Patton et 
al, 1975) pointed up the contribution to use of the "personal factor" which 
1s typified, fcr Instance, when someone takes direct responsibility for 
trying to make use happen. 

Until recently, that "someone" was usually taken to be the evaluator, 
the "provider" of Information. Our research, however, as 1t has amplified 
the "personal factor" and discovered others contributing to use, demon- 
strates that the role of the potential "user" of 1njprmat1on, such as an 
administrator, 1s just as Important as that of the evaluator 1n promoting 
use. In many situations the evaluator him- or herself will lack the power, 
prestige, political sensitivity, or contextual understanding necessary to 
promote use. Our work has shown that use will frequently require the 
Influence of a program administrator who does possess these and other 
attributes. 
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CSE Research on Use 

Drawing on the early studies mentioned above (A1k1n, 1975; Patton et 
al, 1975), we conducted several empirical studies of evaluation use. Among 
these were: (1) evaluation case studies; (2) an evaluator field study; and 
(3) a user survey. These studies contributed to our synthesis of the 
knowledge on use and led to a practical handbook for administrators who 
wish to organize their program evaluations for use. 

The evaluation case studies : The ca.se ^studies (Alkln, Daniak, & 
White, 1979) focused, over a perl odjaf; two tfears, on five different pro- 
grams with required evaluations. These^cases provided detailed descrip- 
tions of^school -level program Implementation and evaluation, and how the 

evaluation process unfolded 1n each program. Our analyses uncovered the 

/ 
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people who shaped the evaluation process', how 1t was used 1n each case, how 
1t fitted 1n with other school operations, and how 1t Influenced 
decisions about the program. Further, by Identifying some of the factors 
promoting these uses, we were able to develop a conceptual framework to 

guide our future study of use. ! 

The evaluator field study : Drawing on the emerging framework, Dalllak 
(1980) spent a year as a participant-observer working closely with three 
school -district program evaluators 1n the belief that observation and 
analyses of evaluators — the providers of Information — at work would 
illuminate conditions of use. By observing these evaluators at work 
Dalllak was able to elaborate some of our previously Identified factors, 
particularly those reflecting the evaluation's organizational setting, as 
well as the kinds of tactics that evaluators adopted to Increase their 
use-enhancing effect. 
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The user survey : The user survey (Stecher, Alkln, & Flesher, 1981) 
took place over the course of a year In 22 schools 1n the district In which 
the field study had previously been conducted. Our concern here was to 

* 

characterize the role of a particular information user, the program admin- 
istrator, In terms of the nature of the decisions' typically confronting 
administrators, and to uncoyerhow and what kinds of Information come to 
shape these decisions. 

The Interviews provided a picture of the kinds of decisions — 
programmatic and other school administrators need to make to do their 
jobs, the ways that they use evaluation and other Information to 
pinpoint a need, to amplify a previous conclusion — as they form these 
decisions, and the broad strategies they adopt to stimulate others to use 
Information 1n their programmatic responslbWtes. 

Synthesis and handbook : To help synthesize the knowledge on use we 
developed an annotated review of the relevant empirical and conceptual - 
theoretical literature, drawn from educational and other settings, (Burry, 
1983), and a handbook for the administrator-user who plans to build use 
Into his or her program evaluation (Alktn et al, 1985 1n press). All of 
our work to this point Illustrated the Importance of user-evaluator ▼ 
collaboration 1n promoting use given various factor Impacts. The handbook 
therefore clusters factors Into patterns which reflect the stages of the 
use process and which can be Influenced to promote use. 
Factors Affecting Evaluation Use 

On the basis of the work described above, we Identified and classified 
the individual factors affecting evaluation use Into three related cate- 
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gorles — human, context, and evaluation procedure or methodology. How 
these factors Interact together determines the extent to Which evaluation 

» 

1s likely to be used. 

Figure 1 lists the three kinds of factors. Those 1n the human 
category reflect evaluator and user characteristics that have a strong 
Influence on use. Included here are such factors as people's attitude 
toward and Interest 1n the program and Its evaluation, their backgrounds 
and organizational positions, and their professional styles. 

Context factors Include the kinds of requirements and fiscal con- 
stralnts the evaluation faces, and the relationships between the program 
being evaluated and other segments of Its larger organization and 
surrounding community. 

The evaluation factors refer to the actual conduct of the evaluation, 
and Include how the evaluator and users work together, the procedures used 
1n the evaluation, and the quality of the Information 1t provides. 

The factors 1n each of the three groups have a demonstrated Importance 
to use, and many of them will require administrative Influence to promote 
use. In the next section of the article, therefore, we offer a series of 
observations drawn from the empirical studies of use. These observations 
help define each of the factors 1n Figure 1 and suggest the kinds of 
Influence they may have, as a precursor to discussion of factor Interaction 
patterns and administrative organizing to promote use as a management 
tool . 

Observations Drawn from Empirical Studies 

With respect to the human factors affecting use, an evaluation's use 
potential 1s likely to Increase to the extent that: 



Figure ii Factors Affecting Use 
!. Human Factors 

A. £ value tor Characteristics 

* \. court tment to use 

2. willingness to Involve users 

3. choice of role 

4. rapport «Hth users 

5. political sensitivity 

6. credibility 
7,. background and Identity 

a. gender 

b. title 

B. User Characteristics 

1. identity 

a. range of potential users 

b. organisational positions 

c. professional experience levels 

2. interest 1n the evaluation 

a. views about the project being evaluated 

b. expectations for the evaluation 

c. predisposition toward the evaluation 

d. perceived need , 

e. percefved risks 

3. eearttment to use 

4. profession*! style 

a. administrative and organisational skilH 

b. initiative 

c. openness to new ideas or change 

5. Information processing 

a* preferences for particular forms r 
b. how Information is processed 

IX. Context Factors 

Pre-ox1at1no Evaluation Sounds 

1. written requirements < 

2. ether contractual obligations 

3. fiscal constraints 

Organisational Features 

1. 4ntraor$aft1tat1oita1 

a. role of central /district office 

b. Interrelationship between unit and central /district 
administration 

c. institutional arrangements 

d. unit level autonomy 

e. sources of Information beyond evaluation likely to be in use 

f . perceived Institutional risk 

2. external features 

a. comuMty climate 

b. community Influence 

c. role of other agencies 

Project Characteristics 

1. age/maturity * 

2. 1nnovat1veness 

3. overlap with other projects 



A. 



C. 



W4i 
./SI 



■'";f 



\ 



ERIC 



C. 



III. Evaluation Factors 

Evaluation Procedures 

1, methods used 

a. appropriateness 

b. rigor 

2. dealing with mandated tasks 
3* used of a general nodal 

Information Dialogue 

1. amount and quality of Interaction between evaluator and users 
Substance of Evaluation Information 

1. Information relevance 

2. Information specificity 

Evaluation Reporting 

1. frequency of Information provided 

2. timing of information 
3* format of presentations 



a. oral presentations 

b. written reports 

c. statistical and narrative data 
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The evaluator — 

• 1s personally committed to peeing his or her work put to use, and 
actively makes efforts to facilitate the use .of Information; 

0 1s willing to Involve users 1n the evaluation through cooperative 
planning and conduct of the evaluation. and Its uses; 

° recognizes that alternative evaluation roles exist, chooses a role 
that 1s appropriate 1n the given setting, and focuses on serving 
program needs and questions in addition to any external 
requirements; „ 

0 develops rapport with users by earning their trust 1n an atmosphere 
of harmony and agreement; 

0 1 s politically sensitive to the program and understands the rel a- 
tlonship among formal and Informal power sources, opinion makers, 
decision making processes, and the function of evaluation as one of 
the Inputs to these processes; 

• establ 1 shes credi b1 1 1 ty i n terms of technical competence and 
personal and professional manner. 

The users — 

• are clearly Identified so that the evaluator understands the range 
of orgarri zati onal posi ti ons and professional experience levels — 
administrative vs. operational , sole or shared decislon-maKing 
authority, familiarity with evaluation which are represented 
among the users and which bear on their potential for using 
information; 

° view the project 1n such ways that they would be willing to modify 
these views, if warranted; 

0 have specific expectations for the evaluation — determining the 
program's efficiency, understanding Its processes, assessing Its 
outcomes — which are translated Into questions and concerns that 
the evaluation will address; 

0 are predisposed to accepting the evaluation's findings, which may 
be because they 

0 have a high perceived need for evaluative answers to their 
questions, and 

0 perceive the risks of the evaluation as outweighed by the potential 
benefits, in addition, they 
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° are personally committed to using evaluation Information as their 
questions and concerns are answered, and 

° have sufficient administrative and organizational skills to act on 
information, to get tnings done. They win 

° take the initiative to use evaluation Information In their own area 
of responsibility and, 1f necessary, to stimulate others to follow 
their example. Further, they 

° are open to new Ideas or change that stem from the findings, even 
if these findings suggest they need to modify their original views 
of the project. And, as the evaluation process unfolds, their 
positive Interest 1n the evaluation remains high, because they 

° ask for and receive the kinds of Information they prefer to use — 
narrative, descriptive, or some combination, through the kinds of 
processes — oral reports, written reports, detailed or summary 
treatments, they are most comfortable or routinely familiar with. 

With respect to the context factors affecting use, an evaluation's 

use potential 1s likely to Increase to the extent that: 

1. The pre-existing evaluation bounds — 

° are characterized by a guided harmony rather than by conflict and 
tension. The evaluation's written requirements legal codes, 
federal/state requirements — permit sufficient flexibility so that 
the evaluator can respond to such other contractual requirements as 
those set by program administrators or operators. 

2. Thg organizational features — 

° are marked by ami cab! e co-ex1 stence 1n an atmosphere stressing 
discussion and the negotiation Of problems and needs; 

° facilitate the central /district office -~ often the evaluation 
sponsor — role in balancing broad system concerns with those of 
the Individual units, such as the schools who are subject to 
evaluation; 

° permit sufficient unit level autonomy so that unit (e.g., a school) 
questions receive a fair share of the evaluator' s attention as he 
or she addresses a variety of broad organizational and unit 
questions of Interest; 

° promote frank discussion of the perceived institutional risks and, 
where there 1s a question of whether the evaluation benefits will 
outweigh the risks, consider the possible outcomes and resultant 
actions the organization might take; 
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° are free from undue or negative Influence from the surround' *q 
community or other agencies , 

3. Program characteristics — 

° are clearly defined on fuch dimensions as age/maturity , 

innovatlveness , and overlap with other programs oecause these 
characteristics have a bearing on the kinds of procedures the 
evaluator should select and the kinds of Information he or she 
should provide 1n order to stimulate use. 

With respect to the eval uatlon factors affecting use, the use 

potential is likely to Increase to the extent that: 

1. The evaluation procedures — 

° are appropriate to the particular project. A selected procedure 
must be appropriate as a method for addressing the given question, 
and also appropriate 1n the context of the project; 

° address the matter of rigor from tbje dual standptot of accepted 
standards of evaluation practice and the users' conception of what 
constitutes rigor; 

° deal with mandated tasks — funding agency requirements, central 
office needs, unit level questions — In- a balanced manner so that 
no single point of view 1s seen to dominate; 

° reflect the viewpoint that no single evaluation model 1s Inherently 
superior; Instead, evaluation 1s seen as a tool for decision making 
and the selection of evaluation procedures Is guided by the 
ded s1 on-making process. 

2. Information dialogue — 

° reflects purposeful , guided sharing of ideas between evaluator and 
users; 

° is ongoing, in sufficient amounts to stimulate or maintain user 
interest in the eval uat1 onTwTfl rgual 1 ty growing out of 
coll eg1 all ty and reciprocity. 

3. Evaluation substance — 

° 1s rel evant from the users' standpoint because 1t constitutes 
pertinent answers to the questions they have raised; and 

° is specific by focusing its content on the needs and interests of 
the particular user or user group. 
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4. Evaluation reporting — 

° 1s marked by frequent and well -focused provision of Information; 

b 1s timely 1n that 1t reflects program chronology and meshes with 
Important events stemming from the program's decision needs; . 

° uses whatevc/ variety of presentation formats — oral, written, 
statistical /narrative, formal or informal — that 1s appropriate to 
the range of users and their evaluation Interests. 

Factor Interactions 

The preceding observations begin to suggest that factors are likely to 
Interact to affect use. Here we win discuss a few possible Interaction 
patterns to Illustrate the kinds of phenomena the administrator might need 
to consider as he or she organizes the evaluation for use, primarily 
because many of the factors are beyond the evaluator' s control. 

For example, to help promote program-level use, the evaluator should 
address questions relevant to the program, questions of Interest to program 
staff. The extent to which the evaluator 1s successful win depend, 1n 
part, on the various requirements for the evaluation, such as those set by 
a funding agency, and whether any particular requirement 1s allowed to 
dominate. But 1t wm also depend on users' Interest 1n the evaluation and 
their comml tment to applying Its "findings. However, users' predisposition ' 
to make this application can be affected by perceived institutional risk, 
pressures from the program's community, and the timing at which reports are 
provided, to mention but a few of the possibilities. 

Many of the factors and interactions suggested above may not be 
amenable to evaluator influence. For example, while the evaluator may 
commit him- or herself to use, the associated user commitment, which also 
contributes to the application of results, is properly 1n the admini- 
strator's sphere of influence. 
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In short, to the extent that the factors mentioned above are subject 
to Influence in a given setting, many are tn the administrator's domain and 
are therefore perhaps mojjB amenable to his or her influence. And this 
Influence, 1f necessary, can cut across all three factor categories, not 
only the context/organizational category traditionally associated with 
administrative responsibility. 

We suggested earlier that to be able to use evaluation as a management 
tool, as a decision-making tool, 1t is critical that those people who may 
be affected by a particular delslon be Involved 1n the decision-making 
process. What do the kinds of Interactions noted above, then, suggest for 
the administrator-organizer trying to Increase an evaluation's potential as 
a management tool? 

First of all, by very virtue of his or her entry Into the use process, 
the administrator becomes one of the factors Influencing use. Continuing 
CSE research on factors promoting high evaluation utilization has suggested 
kinds of evaluator behavior which promote use. These behaviors offer clues 
to the kind of overall demeanor that the administrator-organizer might 
adopt, first of all, to create an atmosphere conducive to evaluation use. 

We recently analyzed several evaluations whose high utilization levels 
were documented as part of an AERA award to recognize such evaluations 
(Ruskus & Alkin, 1984). Many of the factors cited tended to confirm those 
displayed 1n Figure 1. Five of these factors, each of which was cited as a 
use-promoting characteristic, suggest how professional style can have a 
bearing on use. These five factors are level of effort, leadership 
behavior, user Involvement 1n the evaluation, Involvement 1n Implementing 
recommendations, and commitment to use. 
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Evaluation users frequently cited the high levels of evaluator effort 
that contributed to their use of results. From the standpoint of the 
equity theory of motivation (Adams, 1965; Welck, 1966) 1t seems likely that 
when level of evaluator effort 1s deemed to be high, users demonstrate high 
utilization. 

Leadership was another factor cited as contributing to evaluation 
use. Social psychologists such as Ukert (1961) and Stogdlll (1974) 
suggest that leadership may be seen as originating new Ideas; mixing with 
other participants; acting on behalf of other participants; reducing 
conflicts; organizing; communicating; recognizing participants' efforts; 
stimulating participants to achieve; and helping them carry out their 
duties. 

Involving the potential users was another frequently cited factor 1n 
the highly utilized evaluations studied. Beyond the Idea that users are 
likely to use Information when they play a part 1n generating the 
Information, participant management theory (Ukert, 1967) suggests that 
supportive relationships, group decision-making, and shared organizational 
objectives contribute to commitment to carrying out organizational policy 
and decisions. 

Evaluator involvement 1n implementing recommendations also played a 
role 1n the highly utilized evaluations. Such behavior, taking place after 
the report was generated, can run the gamut from Interpreting Implications 
of a recommendation to making concrete suggestions about areas 1n need of 
Improvement and possible means of promoting such Improvement. In this 
vein, sociological theory on the management of change (e.g., Keen & Scott 
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Morton, 1978) suggests that evaluation needs to be concerned with 
Introducing the need for change, striking out 1n the direction of change, 
and Integrating the change Into existing frameworks. 

User commitment to use, finally, seemed Important 1n all the highly 
utilized evaluations. On the basis of themes Identified 1n the marketing 
literature (e.g., Rogers, 1962), the users 1n the evaluations studied can 
be typified as "early adopters" who were (or became) highly disposed to try 
out new Ideas. 

Now, 1n several Important ways, the administrator trying to promote 
evaluation use 1s assuming evaluation-like responsibilities. To the extent 
that such 1s the case, then administrator efforts In promoting use, 1n 
providing leadership to other potential users and Involving them 1n the 
evaluation and 1n Implementing Its recommendations, will help to stimulate 
their commitment to use. That commitment, however, Is likely to be 
short-lived unless the evaluation, or at least part of the evaluation 
effort, 1s focused on user concerns. 
Establishing a User Focus; 

Previously we outlined some functions of evaluation that would enhance 
Its relevance and use as a management tool. We have suggested that, to tpe 
extent an evaluation 1s to serve multiple audiences - funding source, pro- 
gram director and staff — then the needs of these various audiences nee^ 
to be recognized and kept 1n proper balance. To be used as a management) 
tool at the local system level at which the evaluation 1s conducted, the 1 
evaluation must be organized so that, 1n addition to satisfying any other 
requirements, 1t Identifies, addresses, and answers local -level questions. 

co i 
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We mentioned earlier tnat while establishing a local focus presents 
few technical difficulties, 1t may encounter some attltudlnal barriers 
reflecting conflicting Sets of evaluation expectations. For example, let's 
resume the case of the superintendent responding to the Issue of excellence 
1n education. What problems might he or she encounter 1n attempting to 
establish local needs as one of the fod of the evaluation? In attempting 
to get evaluation Information that has local management relevance? 

Consider the kinds of management concerns we alluded to earlier 
(Bursteln, 1984). In terms of drawing upon evaluation to provide Indi- 
cators of progress, who 1s to decide what these Indicators are to be? Are 
there conflicting viewpoints? Can they be reconciled? Can they legiti- 
mately differ 1n various settings? 

In regard to making decisions about students, what kinds of decisions 
are to be made? Are decision needs — such as comparison versus Individual 
diagnosis — 1n competition? Is one kind of measure deemed superior to 
another? Is that viewpoint based In fact or does 1t grow from tradition? 

With respect to program decision making, does the evaluation have to 
address multiple audiences? Are there potential conflicts between them? 
Can the evaluation reconcile external accountability concerns and local 
concerns about program monitoring and Improvement? 

With respect to other possible management applications of evaluation 
-- long-range planning and policy formulation -- 1t 1s unlikely that evalu- 
ation can, or should, be used at the local level unless 1t first has local 
relevance on the other three issues outlined above. Can evaluation come to 
have that local relevance? 
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While the reduction In federal control of evaluation that we alluded 
to earlier 1s Intended to Increase local — SEA and LEA -- responsibility, 
can we assume that the federal intention 1s accepted at these levels? 
Further, can we assume that SEAs and LEAs responding to the possibility of 
assuming greater control of their evaluations will be 1n agreement on basic 
Issues such as evaluation purposes, emphases, and procedures? If not, will 
one point of view dominate and thus reduce the evaluation's relevance for 
the other? What are some of the Issues confronting our school district 
superintendent who wants to make sure the evaluation will serve his or her 
management and decision concerns? 

One primary issue, as we have already suggested, 1s that the evalua- 
tor's ability to focus on one set of needs may be constrained by other 
factors 1n the setting. It may be that one set of needs, requirements, or 
dominant attitudes causes the evaluator to adopt a certain role and collect 
certain kinds of Information which, in turn, may cause the superintendent, 
and his or her colleagues, to view the evaluator and the evaluator's work 
with something less than enthusiasm. We believe that the current situation 
with respect to evaluation fod requires the superintendent's attention. 

First, some of the superintendent's potential evaluation users may not 
be convinced that changes 1n federal requirements will actually reduce 
external supervision and control. That 1s, while federal supervision may 
decrease, the state may continue or Initiate, or be perceived by LEAs to be 
continuing or Initiating, policies which offset LEA attempts to direct 
their evaluations toward LEA matters. 
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Second, some of the superintendent's potential evaluation users may be 
so thoroughly Immersed 1n the business of administering a previously 
required test, perhaps a commercially published, norm- referenced test, 
that they are unresponsive to the possibility of developing a more locally 
relevant test, perhaps a criterion- referenced test of a particular content 

I 

area. / 

Third, decreased federal control 1s accompanied by decreased federal 
funds, and with reductions 1n resources, local school districts may be 
unable to supply sufficient evaluation expertise across the various content 
areas they either need to, or would like to, evaluate. An evaluator may 
need to take responsibility for simultaneous evaluations of Chapter 1 and 2 
programs and of other programs such as bilingual education. 

Related to the above Issues is the emphasis on technical procedures 
that still exist In the regulations accompanying some programs. This 
potential problem area, though It 1s of particular concern 1n bilingual 
programs, 1s seen to some extent 1n the "sustained effect" provision 1n the 
Chapter 1 regulations. Coupled with dwindling local resources and 
evaluation expertise thinly stretched, a school district facing multiple 
program evaluation needs of a technical nature may find 1t difficult to 
comply. 

Further, the "objective measures" mentioned 1n the Chapter 1 require- 
ments may not be uniformly understood. Owing to historical precedent, the 
evaluator of an LEA program may believe that a test must be norm- referenced 
In order to be considered objective and may continue to use this kind of 
test even when it serves no relevant local purpose. If this is the case, 

€3 



the evaluation results are likely to be seen by program ^taff as having 
little practical value for them. ' j 

In another LEA, also concerned with the matter of objectivity, the 

i 

pros and cons of various kinds of tests may be discussed at length without 
ever resolving the matter to the satisfaction of all potential users. So 
much time may be spent on the debate over the test question that the eval- 
uator has little time left for planning and conducting an evaluation around 
more Important Issues. 

In districts meeting the kinds of problems outlined above, that 1s, 
where testing Issues are difficult to resolve, where there 1s limited 
expertise to balance technical adequacy and local relevance, there 1s 
likely to be some negative effect on factors promoting use: anxieties and 
sense of risk may dominate; program staff may believe their questions and 
concerns are receiving Insufficient attention; the evaluator's credibility 
1s likely to suffer; commitment to evaluation use and perception of useful- 
ness will decrease. 

Problems such as those outlined here do appear to warrant the atten- 
tion of the superintendent 1n the case we are using for Illustration. For 
example, we found 1n a recent exploratory study (Burry, 1984) of directors 
and staff members of school district research and evaluation units a 
general agreement concerning the kinds of problem areas described above. 

For example, there was a general concern about the force of historical 
precedent. Because of earlier state and district preferences (which grew 
out of earlier federal emphases), district evaluators were still required 
to administer norm- referenced tests for reporting purposes. But informa- 
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tion produced by such tests was of limited use to them 1n carrying out 
their responsibilities. Further, the evajluators also agreed that these 
tests did not address teachers' Interests or provide the kinds of Informa- 
tion needed to monitor and adjust classroom Instruction. Most evaluators 
felt that a good deal of their limited time and resources were given over 
to generating Information which was state required 1n response to Chapter 1 
regulations. 

A few districts were trying to distill some locally-useful Information 
from a norm-referenced test. But the evaluators felt that too much time 

was Involved 1n having the tests scored and then returned for analysis and 

m 

Interpretation so that they would be of some use to curriculum specialists 
and classroom teachers. Consequently, these efforts detracted from the 
time they needed to generate Information more specific to Instructional 
needs. 

Several districts, 1n addition to administering and reporting the 
results of a norm-referenced test, used district-developed objectives-based 
measures and/or the tests accompanying curriculum materials. (See Burry, 
et al, 1982, for a discussion of how widespread this practice 1s, as well 
as some of its Implications.) While districts found the Information from 
these additional tests 1nstruct1onal1y useful, they asked why they had to 
conduct what was, 1n effect, a parallel evaluation. That 1s, for the 
reasons suggested above, they felt they should use a "respectable" norm- 
referenced test for external reporting purposes; given the limitations of 
the Information provided by such tests, however, they felt at the same time 
that they had to resort to other devices for locally useful Information. 
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The question raised by the evaluators was as follows: 1f such 
district-level efforts are necessary to produce formative data for district 
purposes, cannot these efforts also be used to satisfy external (summatlve) 
requirements? They asked this question even though they were aware that 
federal requirements Impose no particular kind of test; perhaps they (or 
their superiors) continue to believe that anything other than a standar- 
dized test 1s unacceptable. 

This situation created a double dilemma. On the one hand, the 
evaluators realized that, to be locally useful, the evaluation should 
provide different types of Information for different groups of users and 
that such an effort takes time: time to Identify the needs and questions 
of various potential uses, time to develop or select appropriate tests or 
design other data collection procedures, time to win user support for the 
evaluation. On the other hand, the felt need to run a separate evaluation 
for external purposes also takes time, time that might be better spent 
addressing local questions and needs. 

Finally, while some districts were attempting to Increase their 
evaluation's local relevance, the evaluators stated that they had a hard 
time convincing other staff, such as resource specialists and teachers, to 
become Involved 1n the conduct of the evaluation. Because of what they 
knew or believed about previous evaluations, personnel were hesitant about 
raising their own evaluation questions, reluctant to participate 1n the 
process of devising ways to answer them, and unwilling to believe that the 
evaluator would want to help them 1n the task of carrying out' their 
day-to-day responsibilities. 
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In essence, then, then evaluators we Interviewed are describing how 
some potential evaluation users, the kinds of people whose support will be 
needed 1n order for the superintendent to be able to use evaluation to help 
manage his or her district's operations, may act with respect to the 
evaluation, Its conduct, and its uses. They suggest some of the ways the 
factors we Introduced earlier may Interact to affect evaluation use. They 
suggest some likely factor patterns that any administrator will need to 
consider as he or she begins the task of organizing for evaluation use. 

ADMINISTRATIVE ORGANIZING TO PROMOTE USE 

Figure 2, which 1s excerpted from the handbook we described earlier 
(Alkln et al, 1985, 1n press), places the factors that we believe are 
central to use 1n most evaluation contexts into a pattern which will 
facilitate organizing for use. In^hls pattern the factors are grouped to 
reflect stages 1n the process of planning for and conducting an evaluation 
to maximize Its use potential. * 

The factors and their potential Influence on use should be considered 
from the standpoint of the Intended users/uses, gathering Information that 
will help the evaluator focus on these users/uses, and gathering Informa- 
tion that will help the administrator-organizer ensure that factors in the 

setting do not impede that focus. 

The administrator-organizer may want the evaluation to provide 
information that he or she, and other potential users, can apply to one or 
several decision concerns — broad monitoring issues, and/or student or 
program decisions, and/or planning and policy needs. The major tasks, 
then, are to decide on the users/uses on which the evaluation should focus, 



FIGURE 2: 

Factor Pattern For Evaluation Use 
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A. Setting the Stage 



Pre-existing evaluation bounds 



User Identity 



Program characteristics 



\ intra-organizational features 



External features 



B. Identifying/Organizing the Participants 



User interest In evaluation 



User commitment to use 



Evaluator characteristics 
. ♦ background/identity 

• corffitmenttouse 

• willingness to involve user In evaluation 

• choice of role 
•political sensitivity 
•credibility ■ 



Evaluation procedures— plan 



User professional style(s) 



C. Operationalizing the interactive Process 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue—formative 



User information processing preferences 



D. Adding the Finishing Touches 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue — summative 
User commitment to use 
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ERjC Excerpted from Alkin et al. Organizing for Evaluation Use; A Handbook for Administrators , 
Los Angeles: UCLA Center tor the Study of Evaluation, 1983. 
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asking questions 1n light of the situational factors that might Influence 
use, and using the answers to these questions to help guide both the 
evaluation process and the administrative tactics devised to help ensure 
that the evaluation stays on target. 

The administrator who assumes the use-organizing responsibility can 
use the factor pattern 1n Figure 2 with any appropriate emphasis, 
addition, or deletion of factors given the particular context while he 
or she considers the program, its evaluation, the setting in which 1t takes 
place, and the Intended uses. This consideration Involves anticipating the 
effects that a particular evaluation direction, once taken, 1s Hkely to 
achieve. It consists of asking oneself a series of questions 1n light of 
the listed factors with the intention of determining how the program 
embodies each factor; that 1s, deciding whether that embodiment 1s likely 
to have a positive, neutral, or negative effect on the Intended uses, and 
then devising strategies to strengthen or maintain positive effects while 
minimizing negative effects. These strategies may then be Implemented by 
the administrator and/or the evaluator or some other potential user. 

For example, assume that the superintendent 1n our case Illustration 
goes through the process described above and, using the scheme suguc :ted 1n 
Figure 2, asks him- or herself the following question about the first 
organizing issue -- setting the stage: "As part of the intra-organiza- 
tional features, is there any perceived staff risk that might hinder my 
using the evaluation to plan future Instructional offerings 1n response to 
changing student enrollment patterns? Aftei due deliberation and discus- 
sion with potential staff users, the superintendent discovers that some 
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teachers feel that the evaluation poses a risk to them and that the degree 
of risk 1s likely to outweigh possible benefits. Ofcner teachers either 
feel that the evaluation presents Uttle or no risk or that benefits to be 
accrued outweigh any possible risks. 

Now, this user group's reluctance to accept and apply evaluation find- 
ings 1s one manifestation of a two-part problem. First, their acceptance 
and application of Information may be Important to a larger decision area, 
one that may need to be made consensually by all users. Given the reluc- 
tance of one segment of the dedsl on-making group to participate, 1t may be 
that the resultant decision concern 1s never fully resolved. 

Second, those with the sense of risk may advance beyond reluctance to 
participate to outright attempts to convince others of potential dangers. 
If they are successful, then initially receptive users may^later opt to 
remove themselves from the evaluation effort and, further, may attempt to 
thwart the entire effort. 

In such a situation, the superintendent organizing for evaluation use 
would need to ask other questions 1n order to determine: the reason for 
the sense of risk on the part of one user group; whether or not that 
perception 1s justified; the extent to which the group 1n question may 
attempt to convince others of the Imminent risk; the likelihood of 
success. He or she would then need to devise appropriate strategies given 
the answers to the preceding questions. 

For example, 1t may be that the sense of risk 1s unjustified or has 
become magnified, perhaps on the basis of some previous evaluation experi- 
ence. In this situation, the superintendent would need to convince the 
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hesitant group that this perception 1s unjustified so that the evalua tor's 
credibility does not suffer and' the necessary user group Involvement is 
achieved. 

To the extent that the superintendent him- or herself encounters 
difficulty 1n minimizing sense of risk, then 1t may be possible to enlist 
trusted and respected staff members from among the more receptive users to 
help convince their colleagues that, in this particular setting, the risk 
factor 1s unwarranted and that participation 1n the use process is justi- 
fied and Important to the larger institution. 

Keeping the above potentially Inhibiting factor example 1n mind, and 
the kinds of question-raising process and associated strategy formulation 
the superintendent considered, we will now suggest a few possible ques- 
tions, and how they might be addressed, for factors in each of the four 
stages 1n the use process, as depicted in Figure 2. These questions are 
intended to guide administrative organizing for evaluation use, and their 
answers, as with those of all the factors displayed, should Inform the 
administrator's selection of strategies to build use into the evaluation. 
Setting the Stage 

Setting the stage involves determining, before the evaluation planning 
process begins, the kinds of factor Interactions likely to affect use in a 
given setting. While these factors may be set to some extent, they are not 
necessarily "glvens." Note in Figure 2 that this determination considers 
possible effects stemming from the pre-existing evaluation bounds, the 
potential users Identified, program characteristics, and 
intra-organizational and external features. 
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Questions that the administrator-organizer might raise here could 

( 

Include, for example: 

° Who are the Intended users of the evalution Information? 

° Are the pre-existing evaluation bounds such that there may be 
potential conflict, real or perceived, between program expectations 
and other requirements? 

° How 1s the program best characterized with respect to its maturity, 
1nnovat1veness, and overlap with other programs? 

Now, let's narrow the focus a little and add some context before we go 
any further. Suppose that our district superintendent had a programmatic 
concern to resolve. In the district, enrollment 1n math classes 1n some 
high schools has been dropping off sharply 1n the last two or three years; 
1n others, math enrollment 1s staying relatively constant, even increasing 
a little. Board and parental concern with students' "-technical literacy 1s 
on the rise. The superintendent would like the required evaluation of the 
math program to help explain the different enrollment patterns and discuss 
what might be done about It. 

Who might the Interested stakeholders, and hence potential evaluation 
users, be? At the least: the funding agency and the board; building 
principals, math department chairs, math teachers; parents and students; 
district office math specialists. 

What might be some possible conflicts among these users? Should the 
superintendent consider this question 1n light of program maturity/ 
innovatlveness to h*lp illuminate the possibility of conflicting 
expectations? Very definitely. Consider the following: 

0 The board and the funding agency expect the district to continue 
reporting the math program results in terms of student scores on 
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the norm- referenced test that has been 1n use 1n the district for 
the last five years; parents are used to seeing these results 
discussed 1n the local newspaper; some parents want these scores to 
go up; others are asking why their chfldrln did not have to take 
the test. 

The superintendent, after meeting with math teachers 1n both 
reduced- and maintained/Increased enrollment schools begins to get 
the distinct Impression that the district's "math program" does not 
look the same across all schools. In some schools, especially 
those with high enrollment levels, 1nnovat1veness seems to be the 
defining feature. But 1nnovat1veness seems to differ 1n these 
schools. In a school or two, teachers rely heavily on tests they 
have developed themselves to make decisions about Instruction; they 
treat the norm-referenced test as something that has little rele- 
vance for them. In some of the low-enrollment schools, teachers 
stress the importance of the norm-referenced test to their students 
and emphasize its content 1n their instruction. A few teachers 1n 
each kind of school do not fit the general pattern. 
The superintendent would like to explore these differences 1n the 
next year's program evaluation. First, he would like to have the 
norm- referenced test requirement waived for that year. In Its 
place, he would like to conduct Intensive observational studies of 
high school classroom math practice to find out if different 
teacher approaches to math instruction and/or math assessment might 
help explain different enrollment patterns. 

* J 
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0 Will the funding agency accept this plan? W111 the school board? 
How about teachers and parents? If the norm-referenced test cannot 
be waived, could it instead be administered on some sample jbasls fo 
the com|ng year? Might the evaluation apply and analyze ttfe 
norm- referenced test and also conduct the intensive /observational 
study?^/ Will resources permit this? II / 

0 If the/ evaluation is permitted to emphasize the ob'servatljonal J 
compo/ient, and discovers that a certain instructional approach < 

seems to be more effective than others 1n attracting arwf maintain- 

/ 

1ng/ student enrollment, how might the board react to thjs finding? 
Hoj might teachers respond? How might 1t be received fey the 
funding agency? 

Wh1le| questions such as these might all be considered in the initial 
organizing step — setting the stage — some of them may not be resolved 
until later on in the use process. 
IdentlfyinyOrganizing the Participants 

After setting the stage for evaluation planning has taken place, a 
series of questions which amplify user characteristics such as interest in 
the ev&luatibn and commitment to its use, as well as questions reflecting 
relevant evajuator characteristics, should be raised. This process should 
result 1n the! formulation of the evaluator's role and the evaluation 
procedures, carefully matched to users' interests, expectations, and 
professional styles, which will be used. 

Among the! questions that ought to be considered at this stage are: 



Are the Intended users committed to use and, if so, b their 
commitment rhetorical or real? 
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° What do the Intended users expect from the evaluation; are these 
expectations likely to affect their desire or ability to apply 
Information? 

° What would be the most appropriate role for the evaluator to take 
with respect to the program, and will the evaluator be willing and 
able to assume this role? 

° What kind of evaluation procedures will provide the best match with 
users' professional styles? 

Let's pause for a contextual breather again. Continuing the 
superintendent's scenario, what are some of the Issues of concern 1n this 
second organizing stage? 

Let's start by thinking about the evaluator for a moment. Though a 
highly-skilled professional, 1s there anything 1n his or her personal 
comportment that would cause anxiety among teachers whose classrooms were 
being observed? If so, would the evaluator accept a carefully phrased 
suggestion about classroom entry? 

Now let*'s take up a possible pol 1t1co/methodolog1cal problem. Let's 
assume the observational component was sanctioned by the funding agency. 
Let's also assume that the district needs all the resources 1t can get to 
continue Its math offerings. Let's also assume that a well-defined 
classroom practice did seem to account for student interest 1n math. 
Should the superintendent recommend that this particular approach 
implemented d1str1ct-w1de, 1s it Hkely that the agency, although they 
sanctioned the observational study, would find Its results to be credible? 
Is it possible that their understanding of observational data would affect 
their refunding decision? Would the board's reaction to the superinten- 
dent's recommendation be influenced by their financial concerns? How might 
teachers whose practice will be affected respond to the recommended 
change? 
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These kinds of questions need to be resolved before evaluation 
procedures are selected and put into operation. 
Operational 1 zing 'the Interactive Process 

Up to this point, the administrator-organizer has been anticipating 
future evaluation actions and effects; 1n this third stage the carefully 
planned evaluation procedures are put Into effect. The central factor 1n 
this group, execution of evaluation procedures, will temper all other 
factors grouped here. 

Among the questions that should be considered are: 

0 What 1s tne most effective data-collection schedule, and are there 
any possible Impediments to this schedule? 

° Do any of the proposed procedures require any special arrangements 
and, if so, with whom? 

° For each Intended user, what particular kinds of Information and 1n 
what kinds of format will be deemed relevant? 

° What kinds of dialogue, via what techniques, will best match users' 
routine Information processing styles? 

While the evaluation process 1s underway, the superintendent 1n our 
scenario, or any other organizer, would constantly monitor the process. He 
or she would ensure that the evaluation 1s proceeding in light of how 
previously raised questions were answered; determine 1f any unanticipated 
factor influence 1s beginning to emerge; determine 1f an expected Influence 
1s less than anticipated, and 1f resources might be safely shifted to 
another factor of concern. 
Adding the Finishing Touches 

This activity 1s the final phase 1n maximizing the potential for 
evaluation use. The group of factors of interest here represents that 
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point 1n the evaluation process where most, or all, of the evaluation 
Information has actually been collected. That Information must now be 
communicated In such a way that the designated users will actually apply 
the information. 

Among the questions the administrator-organizer should consider here 

are: 

° What combination of written and oral reporting will most enhance 
use of Information? 

° At what tlme(s) should these reports be provided? 

° After the reports are provided, will any final arguments be needed 
to convince users to act on the information? 

Finally, note that the answers arrived at 1n any one stage will 

Influence questions and organizing strategies stemming from a subsequent 

stage. Further, the process is cyclical and permits specifications 

proposed at an earlier stage to be modified (e.g., stressing/ de-emphasizing 

one of the evaluation questions) 1n light of subsequent planning, conduct, 

and emerging receptivity toward the evaluation and Its use. 

CONCLUSION 

We have suggested here that evaluation can serve a variety of educa- 
tional management questions and outlined some of the question areas. We 
have stressed that an administrator's ability to use evaluation as a 
management tool depends not only upon his or her own perception of evalua- 
tion but also on the perceptions of other potential evaluation users in the 
system. Evaluation's Contribution as a management tool is affected by the 
degree to which evaluation comes to be accepted and used throughout the 
various levels of the system. 
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Research has uncovered a variety of factors Influencing an evalua- 
tion's use potential. These factors reflect human considerations — such 
as people's attitudes toward and expectations for the evaluation; context 
considerations — such as an evaluation's requirements within a particular 

setting; and evaluation considerations — such as procedures used and means 

\ * , 

of communicating Information. The research demonstrates that an evalua- 
tion's use potential, and therefore its application as a management tool, 
can be greatly enhanced 1f someone takes responsibility for organizing the 
evaluation to meet sped fled \eeds, for particular users, 1n light of the 
factors operating 1n the given setting. The research also suggests that a 
program-level administrator 1s 1n\he most strategic position to assume 
this responsibility. \ 

It seems evident that adm1n1strators\and evaluators must come to know 
more about each other's operational needs and viewpoints. To the extent 
that administrators and evaluators share responsibility for setting an 
evaluation's fod and purposes, and ensure that the evaluation addresses 
these purposes, the evaluation's decision-making power and relevance are 
Increased. * 

When.* program evaluation is being considered, therefore, the 
administrator .needs to decide on Its various audiences, determine their 
questions and information needs, and anticipate the uses they are likely to 
make of the Information. At the same time, the administrator needs to 
consider the factors existing 1n the given setting that are likely to 
Influence these uses. The framework we suggest can be applied to organize 
the evaluation so as to have a high potential for meeting the Intended uses 
in light of various factor Influences. 
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This paper reports on the factors that characterize 
high-utilization evaluations. It is based on materials submitted 
to an AERA Division H competition that was instigated and 
organized by the Evaluation Productivity Project of UCLA's Center 
for the Study of Evaluation (CSE). This project, which is funded 
by the National Institute of Education (NIE), has been at the 
forefront of research on evaluation utilization. 

The paper is organized into three sections. The first 
section outlines the background of the study: the purposes and 
procedures of the Division H competition, and the conceptual 
framework used in analyzing the data. The second section 
describes the analytic* methods, summarizes the results of the 
analysis, and specifies the six factors that seem to distinguish 
evaluations whose results are used by decision-makers. The final 

section profiles the evaluators themselves. The data are viewed 

r 

and interpreted from a multidisciplinary perspective which draws 
upon theories from psychology, sociology, organizacional 
behavior, management, and marketing. 

BACKGROUND 

The primary purpose of the Division H competition was to 
recognize and reward excellence in promoting evaluation use. Its 



-1- 

86 



secondary purposes were to make both evaluators and potential 
users more use-oriented and to encourage practitioners to engage 
in those kinds of behaviors that seem to promote evaluation 
utilization. 

Competition Procedures 

Evaluators entering the Division H competition were required 
to submit an application form (see Appendix A) in which they 
discussed those features of the evaluation that, in their 
judgement, had contributed significantly to its ultimate 
utilization. Evaluation utilization was verified by means of 
materials submitted by evaluation users, who were asked to 
indicate the extent of use and to specify those aspects of the 
evaluator's performance that had contributed to use (see 
Verification Form in Appendix B). Over 30 users (administrators 
of the target programs) submitted such materials. 

Twelve evaluation studies were entered into the 
competition. While all of them dealt with educational programs, 
the majority (seven) were directed to public school programs at 
the district level. Of the remaining five, three addressed 
university progrns; one, a statewide educational program; and 
one, a vocational education program for a special-needs 
population. Three of the twelve studies focused specifically on 
teacher effectiveness. Most of the evaluators had been retained 
as private consultants, although four were employees of the 
organizations housing the program being evaluated. 
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The review process involved over twenty judges, all of them 

JS. 

Division H members, who were selected either because they had 
contributed to tile literature on evaluation use or because they 
were among the leading evaluation practitioners. Bach evaluation 
study, along with the documents submitted as supporting evidence, 
was reviewed by three judges, who rated applicants' on four 
dimensions: (1) conception of evaluation use, (2) extent of 
evaluation use, (3) degree of direct link between the evaluator *s 
efforts and subsequent use, and (4) uniqueness/creativity of the 
evaluator 's efforts to promote use (see Reviewer Rating Form in 
Appendix C). In addition, to ensure against rater bias, each 
rater ranked the applicants whom she or he had reviewed. Ratings 
were then totaled, and from these total scores, four finalists 
were identified; a fifth finalist who had received relatively 

J" « 

high rankings was added to this group. The competition winner, 
and two honorable mentions, were chosen by a subset of raters at 
a full discussion session of the 1984 JRA convention. 

Conceptual Framework 

According to the definition employed in this paper, which 
was developed by Alkin (1975) ar i Patton et al . (1978), the term 
"use" is not limited to the direct application of evaluation 
findings to a specific decision but rather refers to the broader 
supporting role that evaluation findings often vay in the 
complex, ongoing process of decision-making. As Alkin (1982) 
points out, the best definition for utilization is one that 
recognizes the gradual, incremental influence of evaluation. 
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This view of utilization was further developed by CSE through a 
series of evaluation case studies (Alkin, Daillak, & White, 
1979), an evaluator field study (Daillak, 1980), and an 
administrator user survey (Stecher, Alkin, & Flesher, 1981). 

On the basis of these empirical data, Alkin and his 
colleagues (1985, in press) have developed a "utilization 
framework" that classifies the factors affecting evaluation use 
into three interrelated categories: human factors (evaluator and 
user characteristics); context factors (pre-existing evaluation 
bounds, organizational features, and project characteristics); 
and evaluation factors (procedures, information dialogue, 
substance of information, and reporting). Within each of these 
factors, specific elements that influence utilization have been 
identified. For example, critical evaluator characteristics 
include commitment to use, rapport with users, and credibility. 
Critical user characteristics include interest in the evaluation, 
commitment to use, and information-processing preferences. 
Pre-existing evaluation bounds, a context factor, encompasses 
written requirements, contractual obligations, and fiscal 
constraints. Evaluation reporting, an evaluation factor, 
includes the frequency and timing of reports and the mix of 

4,'' 

r 

statistical and narrative data in reports. Many other elements 
are subsumed under the various factors (for a complete listing, 
see Alkin et al. (1985, in press)). 
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ANALYSIS OP APPLICANT DATA 

The comments of users and evaluators were analyzed by 
reading through ' the application and' verification materials and 
abstracting all; statements pertaining to use. These statements 
were then classified according to. the three categories of 
utilization factors: human, context, and evaluation. (User data 
are summarized in Appendix D, «and evaluator data in Appendix E. ) 
In some instances, the results confirmed the importance of the 
factors already specified in the utilization framework described 
above. In other cases, the results suggested that certain 
factors not explicitly identified by previous research need to be 
further explored, with a view to elaborating the framework. 

Almost half of both the users' and the evaluators' comments 
pertained to human factors, and nearly all of the remainder 
referred to evaluation factors. Context factors were rarely 
mentioned, probably because the competition emphasized the 
evaluator 's contribution to enhanced utilization. 

The most frequently cited evaluator characteristics were 

choice of role and willingness to involve users in the 

evaluation. The users' interpretation of choice of role is a 

unique one and will be discussed in more detail below* Also 

important were the evaluator 's credibility and rapport with 
users . 

The evaluation factors] most frequently mentioned were 

/ 
/ 

procedures and reporting. Bpth users and evaluators recognized 
the importance of sound methodology, user-oriented designs, and 
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follow-up procedures. Of the several reporting features cited as * 
contributing to evaluation utilization, a good balance between 
statistical • and narrative data was seen as most relevant. 

While the factors cited by evaluators tended tfo confirm the 
utilization framework already developed, users' comments 
suggested some additional* factors that may influence 
utilization. Because of their significance from the users' 
perspective, six factors merit further discussion for the insight 
they give into utilization: 

o Level of evaluator effort 

o Leadership behavior 

o User involvement 

o Evaluation reporting 

o Evaluator involvement in implementing recommendations 
o User commitment to use 

Level of Evaluator Effort 

Users frequently said that the exceptional level of effort 
which the evaluator put into the evaluation contributed to their 
utilization of the results. The following comments illustrate 

this point: * 

Tho evaluator ( E ) has been willing to expend time and 
energy beyond the typical work day to do the necessary 
wor k . 

The sustained efforts of E over the last three years 
have insured that there is a broad commitment to this 
evaluation. 

E's drive to see the project through and willingness to 
reach out and help contributed to making the evaluation 
useful • 

O t . -6- 
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Clearly, users appreciate the effort put forth by the 
evaluator, not only because it results in a higher-quality 
evaluation and a better evaluation report but also because it 
signifies a commitment to the evaluation and a concern about its 
outcomes. It would seem, then, that when the evaluator manifests 
a high level of energy, the likelihood of evaluation use 
increases* ! 

i 

The equity theory of motivation (Adams, 1965? Weick, 1966) 
suggests that something besides simple appreciation is at work 
here. This social comparison theory views human relationships as 
transactions involving inputs and outputs. In negotiating 
exchanges in the work environment, people seek a balance between 
inputs and outputs. According to this theory, the evaluator 's 
level of effort constitutes an input, and utilization of the 
recommendations advocated by the evaluator constitutes an 
output. Therefore, when level of evaluator effort is. judged to 
be high, users are motivated to demonstrate high utilization. 

Leadership Behavior 

A number of the evaluator characteristics noted by users as 
contributing to utilization can be characterized as leadership 
behaviors. And while some of these behaviors are already 
represented in the utilization framework (i.e., choice of role), 
it seems reasonable to restructure the framework slightly by 
subsuming these elements under the "leadership behavior" rubric. 
Such an alternative conceptualization may provide additional 
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insights into the dynamics of evaluation utilization. 

A sizable portion of the management literature deals with 

the definition of leadership and of what constitutes leadership 

behavior (see Fiedler, 1967; Hollander, 1978; House & Baetz, 

1979; Katz & Kahn, 1978; Vroom & Yet ten, 1973). Perhaps the mpst 

thorough of these discussions comes from social psychologists 

like Rensis Likert (1961) and Ralph Stogdill (1974), who have 

identified nine dimensions of leadership: initiation (originates 

new ideas and new practices), membership (mixes with the group), 

representation (acts in behalf of the group), integration 

(reduces conflicts between members), organization (structures the 

work of members), communication (provides information to members 

and shows awareness of affairs pertaining to the group), 

recognition (expresses approval of group members), production 

(sets levels of effort for greater achievement), and 

consideration (helps members and explains procedures). These 

dimensions were epitomized in the actions of evaluators, as 

reported in a number of statements made by users: 

E's information caused us to initiate actions that 
might never have been taken without his impetus. 
{ Initiation) 

E has the ability to develop rapport and trust with key 
program personnel. This trust is reflected in the way 
those she interviews and surveys open up to her with 
total confidence. (Membership) 

The information E has been able to provide for us was 
just the documentation we needed to justify program 
decisions. (Representation) 

E worked with program personnel and upper 
administration to resolve conflicts, facilitating a 
process where all points of view were heard. 
( Integration) 
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The evaluation report contained practical management 
plans based on the findings. (Organization) 

E's evaluation report was primarily useful in telling 
our story. He was capable of making explicit our 
commonly held (but taken for granted) assumptions about 
what we do and why we do it. (Communication) 

E has provided the impetus for the development of goals 
and objectives to strengthen and improve programs. 
(Production) 

I provided the staff directly connected to the program 
. . . with much positive feedback. (Recognition) 

E explained what could and could not be measured , what 
instruments would be used and why. (Consideration) 



User Involvement 



The evaluator's willingness to involve users — already 

identified in the utilization framework as an important evaluator 

characteristic — deserves further examination because of the 

frequency with which it was mentioned, both by users and by 

evaluators, as a determinant of utilization. Users expressed the 

importance of their own involvement as follows: 

As superintendent, I was involved in the total process 
enough to follow through on the report. 

E developed "ownership" from the beginning of the 
evaluation process so that participants actually 
thought of it as "their" evaluation. 

During planning, E reached out to program participants 
for their involvement, assuring broad-based ownership 
of the entire evaluative process. 

It makes intuitive sense that users will be more likely to 
use information that they asked for or played a part in 
generating, but is there more to it than that? Participant 
management theory (Likert, 1967) maintains that user involvement 



is a critical component of effective management. Participant 
management is characterized by supportive relationships, group 
decision-making, group methods of supervision, and organizational 
objectives that reflect the needs- and desires of all shareholders 
in the organization. According to its proponents, not only does 
this method of management make for better decisions, but it also 
guarantees that people will be more committed to carrying out 
these decisions. Evaluators who strive to build rapport with 
users, who involve users in the design of the evaluation and in 
data collection, and who^consider the information needs of all 
users are following-'^the principles oj participant management, 
whether they are aware of it or not. While this theory validates 
.the* inclusion of users in decision-making, it does not really 
explain why this technique is so powerful. 

« 

One possible explanation is that involving users in the 
process of evaluation changes their attitudes about what the 
program should be and how it should operate. This new attitude 
is reflected in the recommendations that emerge from the 
evaluation. Thus, acting on the recorumenda tions is consistent 
with their attitudes, whereas failure to use evaluation findings 
is likely to result in a state of cognitive dissonance 
(Festinger, 1957) which must eventually be resolved. User 
involvement is really a technique for attitude change that, once 
accomplished, motivates behavioral change. 



Evaluation Reporting 

# While the utilization framework identifies several elements 

of evaluation reporting — notably, content of reports (Alkin, 

Daillak, & White, 1979) and style (Brown, Braskamp, & Newman, 

1978) — as critical to utilization, users in the Division H 

competition noted another critical reporting element: 

thoroughness. This element is implicit in several of the 

utilization framework factors, being most closely related to 

substance of evaluation information, which includes information 

specificity. Nonetheless, users' views on the importance of the 

evaluator's thoroughness in reporting merit further examination. 

The following comments are illustrative of these views: 

The depth and thoroughness of the first-year evaluation 
enabled me to spot quickly and accurately the problems 
with the project so that they could be remedied the 
next year. 

The thoroughness and detail on what went well, what did 
not, and why, gave me a wealth of information and 
insight which I could utilize, in detail. 

This preference for detailed information, rather than just 
summaries and generalities, underscores an important point. 
Evaluator3 must demonstrate their thoroughness, but at the same 
time, they must know the individual users well enough to tailor 
their presentation of information to the cognitive styles and 
preferences of the users. The users quoted above obviously 
prefer that full and precise data be included within the report. 
Other users, with less of a need for fine detail, prefer a 
summary of evaluation highlights. Evaluators must satisfy both 
groups by providing comprehensive but readable reports along with 
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concise executive summaries . They should at all times preserve 
the impression that comprehensive data are available as back-up. 

Style and format are two dimensions of evaluation reporting 

specifically included within the utilization framework. Report 

style is the manner in which the evaluation "message" (i.e., the 

findings) is executed. Marketers have identified several 

successful message execution styles: slice-of-life , mood or 

image, technical expertise, scientific evidence, and testimonial 

evidence (Kotler, 1980). The users in this sample tended to 

prefer styles that were literate, conversational, warm, and 

down-to-earth rather than overly formal and jargonistic. One 

user commented: 

E had the ability to translate facts and figures and 
charts that would normally baffle the layperson's mind 
into interesting material. 

Another noted: 

All of the numbers, statistics, data were surrounded by 
literate prose with appropriate quotations from Alexis 
de Tocqueville. 

Format, another component of message execution, can also 
make a difference in message impact. Format elements include the 
size and dimensions of the report, and its use of color, 
illustration, and other graphic elements. The importance of 
format was acknowledged by many users. The following comment is 
typical : 

E's reports have been exceptionally well accepted. Her 
formatting and special touches, such as attractive 
customized covers and pertinent cartoons, make the 
reading enjoyable and interesting. 



Evaluator Involvement in Implementing Recommendations 

Users' comments also focused on the evaluator 's active role 

in the actual , hands-on implementation of the study 

recommendations. These evaluators went well beyond the 

conventional role of encouraging utilization — they made certain 

of it. The following statements illustrate this proactive „ role : 

E took leadership and responsibility during discussions 
and in preparing proposals and plans for the Board of 
Education. 

E has been remarkably successful in working with 
academic units throughout the campus in 
institutionalizing outcome information on an ongoing 
basis. 

E has held workshops for teachers and staff and is 
always available to help solve problems. 

» « 

Clearly , evaluator involvement in implementing 

recommendations may take many forms. In some instances , the 

evaluator conducts workshops on the findings, as a step in 

potential implementation. Or the evaluator may interpret the 

action implications of particular recommendations. A similar 

phenomenon occurred in a study of Title; I evaluators (Alkin, 

Stecher, & Geiger, 1982): One evaluator helped to attain 

utilization by "suggesting changes in the program, planning next 

year's workshop, developing a dissemination plan, and creating a 

» 

meaningful attendance policy" (p. 2). In essence, this evaluator 
trained school district personnel to use data in making school 
decisions by giving them practice with the process. 

Sociological theory on the management of change provides 
insight into how post-report evaluator behavior influences use. 



98 



Keen, and Scott Morton (1978) hold that the change process 
consists of three stages: unfreezing, which serves to disturb 
the current stable equilibrium and introduces the need for 
change; moving, which involves striking out in a new direction; 
and refreezing, which requires integrating t^he change into 
existing behavioral frameworks to recreate a whole, natural 
entity. Management theorists (Katz & Kahn, 1978) claim that, 
while many change agents are successful in the first and second 
stages, .they fail to realize the critical importance of the last 
stage. The evaluation process can be viewed as the first stage, 
the evaluation recommendations as the .second stage, and the 
implementation of recommendations as the third stage. It is no 
wonder that eyaluator involvement in this final stage of the 
change process characterizes high-use evaluations. 

User Commitment to Use 

User commitment to use, which was identified as ah important 

user characteristic in all of our prior work, also turns out to 

be important in this study. Although users themselves did not 

often cite , this factor — perhaps because they were asked to 

focus on the evaluator in their documentation — evaluators 

referred to it frequently, as the following comments show: 

The people involved were ready for the evaluation and 
the changes. 

The leadership of the Chancellor and the Provost • . . 
was one of the unique factors tha-t contributed to my 
success in promoting the use of evaluation data. 
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To some extent, commitment to use is a personality 
characteristic of users and thus is already established by the 
time the evaluator enters the scene. The marketing literature 
(Rogers, 1962) places consumers on a continuum, depending upon 
their willingness to adopt new products or new ideas: early 
adopters, early majority, late majority, and laggards. The users 
in this sample were clearly early adopters. Given their 
predisposition to try new ideas, they may have utilized any 
evaluation findings they were given. On the other hand, our 
earlier research (Alkin, Daillak, & White, 1979) shows that the 
actions of the evaluator can positively influence the users' 
predisposition to use. 

PROFILE OF A HIGH-UTILIZATION EVALUATOR 

This study provides some insight into the characteristics 
and behavior of those evaluators whose work has a high 
probability of being utilized. High-utilization evaluators have 
personality f traits — concern, warmth, patience, integrity, 
openness, tact, willingness to listen — that make them 
attractive to users from the outset. But they are also true 
leaders. Motivated by the desire to see their work utilized, 
they systematically . enlist the participation of all potential 
users. They often violate . the conventional image of the 
evaluation consultant by becoming involved in program operations 
and program improvement. In all of their leadership activities, 
however, these evaluators remain conscious of users' concerns and 
problems. They freqtrsntly assume the role of teacher, coaching 
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program personnel in evaluation techniques* Their enthusiasm is 
not lost on users, who perceive such evaluators as investing a 
great deal of effort in the evaluation process and as being truly 
concerned about improving the program being evaluated* 

« 

When preparing the evaluation report, high-utilization 
evaluators are careful to respond to the cognitive styles of the 
various user*, and to translate quantitative data into interesting 
information about the program. Once the evaluation report has 
been delivered, they continue their active involvement in the 
program by providing specific guidance in the implementation of 
the recommendations. 
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Appendix A Application Form 

AERA DIVISION H 
• AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 



Evaluator: 



Program evaluated:^ 



School distr1ct/0rganizatipn:_ 

Address: 



1. Briefly describe the program you evaluated, 



» 

2. Whatxuse was made of the evaluation information which you produced? Please 
describe in detail and provide documentation of use, if available. Do not 
send the evaluation report itself. 

V 



3. What were the unique factors in this evaluation contributing to making use 
happen? 



\ 
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Appendix B Verification Form 

AERA DIVISION H 
AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 



Dear 



Your evaluator, ' , who worked on the evaluation 

of • !» is a nom i n e e ^ or an AERA 

~ (name of program or project! 

award for excellence in the area of promoting evaluation use. We would appreciate 
your comments to assist our panel 1n judging the performance of this nominee on the 
above-noted program evaluation. 

1. In what way or ways was the evaluation information provided to you about the 
above program useful ? 



2. In your judgment, what aspects of the evalua tor's own performance during the 
evaluation helped make the evaluation' useful to you? 



< 



Mail this form directly to: Name:. 



Dr. Marvin C. Alkin ■ Position:. 

Graduate School of Education * 



Center for the Study of Evaluation \ D a te: _ — 

UCLA 

Los Angeles, CA 90024 

M 

ERiC P1ease return by March 1, 1984 . 1Q& 



Appendix C 

AERA DIVISION H 
AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 

REVIEWER RATING FORM 

Please rate tha applicant's submission, including documentation and validation, on 
the following criteria. Use the corroborating evidence 'Submitted in support of the 
application to assist your judgments. 

Conception of evaluation use : 

1 the applicant showed little or no understanding of evaluation us, 

J 

3 the applicant showed an acceptable level of understanding of evaluation use 

4 

5 the applicant showed full understanding of evaluation use 



Degree of evaluation use : 

s 1 no indication of evaluation use 

3 some indication of evaluation use 

— — — — — *• > 
4 

5 strong indication of ^valuation use 
— — — j 

Degree of direct link between evaluator efforts and subsequent use : 

1 no link was demonstrated between the evaluator's efforts and subsequent use 

2 

3 a minimal link was demonstrated between the evaluator's efforts, and subsequent use 

4 

5 a strong link was demonstrated between the evaluator's efforts and subsequent use. 

Uniqueness/creativity of evaluator's effort at promoting use : 

1 the evaluator showed no creativity or originality of effort 

2 

3 the evaluator showed a minimum of creativity or originality of effort 

4 

5 the evaluator showed a significant amount of creativity or originality of effort 
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Appendix C ( cont ' d . ) 



5. Please rank thfe entries you have read . 
Rank j Entry No. 



ERIC 



Name of Reviewer 
Date completed 



Appendix D 

Frequency of User Comments by Utilization Framework Categories 



Category 

Human 
Factors 



Context 
Factors 



Evaluation 
Factors 



Factor 



Evaluator Characteristics 



Total Category Frequency 
Project Characteristics 

Total Category Frequency 
Evaluation Procedures 

Information Dialogue 



Element 



Commitment to use 
Willingness to involve users 2 
Choice of role — leadership 
Rapport with users 
Political sensitivity 
Credibility 

Background and identity 2 
Leve^oi- perceived effort 



Innovativeness 



Methods 

Use of a general model 

Amount and quality of interaction 



Frequenc y 



3 
6 
13 
7 
5 
9 
6 
6 

55 



18 
1 



V 1 



Substance of 

Evaluation Information 



Information relevance 
Information specificity 



9 
7 
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L These data are based on the comments of 34 users. 

J This variation of the existing factor was newly discovered in this research 

' 10.9 
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- — Appendix D 



Category Factor 

Evaluation Evaluation Reporting 
Factors 
(cont'd.) 



Total Category Frequency 



Total Frequency 
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(cont'd. ) 



Element Frequency 



Frequency of information provided 1 
Timing of information 3 
Style of oral presentations 1 
Format of reports 3 
Mix of statistical/narrative data 6 

55 
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Appendix E 

Frequency of Evaluator Comments by Utilization Framework Categories 



Category Factor Element Frequency 

Human Evaluator Characteristics 

Factors Willingness to involve users 7 

Choice of role 3 

Rapport with users 3 

Political sensitivity 2 

Credibility 2 

User Characteristics 

; Interest in the evaluation 2 

Commitment to use 4 

I Professional style 1 

Total Category Frequency 24 

Context Organizational Features 

Factors External features 2 ^ 

Total Category Frequency 1 

Evaluation Evaluation Procedures 

Factors Methods 11 

Use of a general model 1 

Information Dialogue 

Amount of interaction 1 

Substance of 

Evaluation Information Information relevance 4 

r Information specificity 1 



These data are based on the comments of 12 evaluators. 
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Appendix E (cont'd.) 



Category 

Evaluation 
Factors 
(cont'd.) 



Factor 



Evaluation Reporting 



Element 



Frequency of information provided 

Timing of information 

Style of oral presentation 

Format of reports 

Mix of statistical/narrative data 



Frequency 



2 
1 
1 
2 
3 



Total Category Frequency 



27 



Total Frequency 



53 
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The Evaluation Productivity Project, scheduled to/run 
through November 1985, was initially intended to explore^ the 
relationship between costs and the extent of utilization of 
evaluation. The original plan called for the development\of a 
set of case scenarios during FY 1984, to be used in the empirical 
research of FY .985. Each of these scenarios was to be "costed" 
in terms of its direct and indirect costs. Subsequently, the 
plan was alteredXto give greater, emphasis in FY 1985 to 
additional synthesis x *->^n4__J&««em4nation of the project's 
longstanding research on evaluation utilization. This change in 
emphasis^ > >a4cJlg_^ith the decision not to undertake new empirical 
research, rendered \the planned costJing exercise pointless. 
Nonetheless, it seems abpropri ate to /set down here some of the 
reflections on evaluation costs that emerged during the initial 
stages of work. (See -alsVx ^Afikin & Solmon, The Costs of 
Evaluation , 1983.) 

This paper, then, summarizes our views on the costs of 
evaluation and, in particular, our "hunches about the kinds of 
costs associated with factors known to affect utilization. The 
first section deals with general issues involved in identifying 
and valuing cost components. We will then discuss the 
appropriate costs of evaluation and consider costs in relation to 

\ 
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benefits. The final section spells out the cost implications of 
high-utilization factors. 

Identifying and Valuing * Evaluation Costs 

Before we consider . how the costs of evaluation can be 
identified and valued, a distinction must be made between direct 
and indirect costs. "Direct costs" are simply the direct cash 
outlays necessary for initiating and implementing the 
evaluation. Examples include salaries, travel expenses, and the 
cost of test booklets and data processing. Because direct costs 
are easy 1 to identify, they are often regarded as synonymous with 
the costs of evaluation. 

la ■ '■ 

Like other educational program activities, however, 
l - evaluations involve indirect as well as direct costs. While our 

discussion of costing focuses on the direct costs of evaluation, 
some consideration of the definition of indirect costs, and of 
general ^issues related to their identification and inclusion in 

the overall "cost" package, seems warranted. 

*• 

Most authorities* (Alkin & Stecher, 1983? Catterall, 1983; 
Haggart, 1983; Morell & Weirich, 1983; Sanders, 1983; Solmon, 
1983) recognize the \mportance of t°hese less obvious costs. But 
there is confusion irt the literature over exactly how indirect 
costs should be Refined. As;a jcesult of both this ambiguity and 
also their tendency to be leir^vlsible , indirect costs are often 

■ r 

overlooked or underestimated when the costs of evaluation are 
computed . 
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"Indirect C03ts" can be defined in one of three ways. 
According to one point of view (Morell & Weirich, 1983), indirect 
costs are those dollar costs not specifically incurred by the 
evaluation project but shared across projects within the larger 
organizational structures clerical time, facilities, 
communication, and even, in some instances, the salaries of 
project staff. This definition is probably the most 
straightforward. Indirect costs of this type are often 
calculated as a percentage of the direct costs of the evaluation 
and are included as a separate line item in the overall budget. 

According to a second point of view (Catterall, 1983), 
indirect costs are the opportunity costs of the evaluation: the 
use of equipment that could be used in other ways; volunteer time 
that could be spent on other projects? pupil time that could be 
spent in learning instead of -est^g? and the time the state 
agency spends in legislating, developing, and monitoring the 
evaluation. Almost every party involved in an evaluation could 
be engaging in other, possibly more valuable, activities. Thus, 
the opportunity cost is the best alternative use of a resource 
used in the evaluation. Interpreted in this way, indirect costs 
can be included in the cost package as the dollar equivalent of 
alternative uses of evaluation resources, though Catterall 
suggests that these indirect costs are often better presented in 
their natural units (for instance, as the number of hours 
students could spend learning) rather than their dollar 
equivalents . 
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Yet a third point of view (Solmon, 1983f^ holds that indirect 
costs are the side effects of the evaluation. For example, when 
a project concentrates on achieving immediately observable 
objectives in anticipation of an evaluation, certain long-term 
project goals may be slighted or even sacrificed altogether. 
Thus, the evaluation can be said to entail indirect costs. (It 
should be noted that Solmon considers opportunity costs 
separately from indirect costs.) * 

As mentioned above, indirect costs in the first sense — 
shared costs of projects within an organization — are relatively 
easy to calculate, since they represent dollar costs for 
materials, personnel, and so forth. Calculating indirect costs 
in the second sense — opportunity costs — is somewhat more 
difficult, since one must determine alternative uses of resources 
and their corresponding values. It is indirect costs in the 
third sense — as side effects — that are most difficult to 
calculate; and we are not aware of systematic attempts to include 
such indirect costs as part of a total evaluation cost package. 

Several authors (Catterall, 1983; Haggart, 1983; Levin, 
1983) have outlined the procedures necessary to identify the 
costs associated with an evaluation. According to Haggart, the 
process begins with a definition of the scope of the evaluation. 
The scope depends on the extent of the education intervention, 
the level of decision-making involved (the higher the level, the 
greater the scope of the evaluation), the purpose of the 
evaluation, and the complexity of the evaluation design. These 
aspects of the evaluation provide the context within which 
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appropriate cost categories may be identified. 

Most authorities agree on the specific cost categories to be 
employed. For example, the cost category systems presented by 
Alkin and Stecher (1983), Morell and Weiricb (1983), and Sanders 
(1983) are highly similar, differing only in detail. Drawing on 
the input of a large sample of professional evaluators, Alkin and 
Stecher (1983) delineate seven typical cost categories: 
professional staff, clerical staff, external consultants, 
materials/supplies/telephone, data processing, facilities, and 
travel. Similarly, Morell and Weirich (1983) break down 
evaluation costs into the following categories: personnel 
(evaluation staff, consultants, program staff, and subjects), 
technological capabilities (data processing, telephone services, 
and duplication) travel, office space and furnishings, supplies, 
and dissemination of findings (graphic artists, printers, and 
audiovisual specialists). The most detailed listing of cost 
vcategories is that provided by Sanders (1983): evaluation staff 
salary and benefits, consultants, travel and per diem, 
communication, printing and duplication, data processing, printed 
materials, office supplies, subcontracts (outside of consulting), 
and overhead. Sanders expands on the types of costs frequently 
involved in data processing: systems design, data coding and 
checking, data storage and retrieval, computer programming, 
computer use for manipulating or analyzing data, and 
computer-based bibliographic searches. 

After the cost categories have been identified, the next 
step is to determine values for each of these categories. One 




method for valuing "ingredients" is described by Levin (1983), 
who offers the following hints about costings 
Personnel 

Salaries and fringe benefits (percentage of 
salaries)} - Value determined .by marketplace 
prices. When personnel costs cannot easily be 
ascertained, use estimates of market value for 
similar services. 

Facilities 

Rented/leased facilities: Value determined by 
annual cost of expenditure. 

Owned facilities: Value determined by cost for 
similar space or annual cost (taking depreciation 
and interest on remaining undepreciated value into 
account) . 

Equipment 

Purchased: Value determined in the same way as 
for facilities, depending on whether leased or 
owned. 

Donated/Borrowed: Value determined as cost of 
leasing or renting similar equipment. 

Supplies 

Value determined by adding estimated expenditures 
to estimated value of contributed supplies. 

Client Inputs 

Service: Value determined by total expense 
associated with service or by market price of 
service provided. 



Note that these valuing procedures can be adapted to any of 
the cost-category frameworks described above. 
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Using a variation of Levin's ingredients approach, Catterall 
(1983) offers an analysis of the costs of testing that can be 
generalized to evaluation* The first step is to do a cost 
inventory. Again, the cost-category frameworks described above 
constitute alternative ways of completing the cost inventory. 
The second step is to total the costs, which involves determining 
the actual monetary figure for each of the cost categories. 
(Levin's procedures outlined above provide a methodology for this 
step. ) The last step in Catterall 's process is to locate the 
cost: that is, to decide who will pay for a particular component 
— the sponsor, a government agency, private party, the clients 
or subjects. An examination of Catterall's variation of the 
Levin Ingredient Chart' (Levin, 1975; see Figure 1) provides 
insights into the distributed features of evaluation costs. This 
chart is particularly informative with respect to indirect 
costs. Note, for example, that most of the cost-associated 
columns (particularly columns four, five, and six) will usually 
list indirect costs, although there may be some direct costs 
related to contributed private inputs. The last row in the 
chart, client time and other client inputs, presents a set of 
costs, all of which are likely to be indirect. In addition, many 
facilities and equipment costs would be considered indirect if 
accounted for in the costs of conducting evaluations.- 
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Entity Bearing Costs 
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( 2 ) 


( 3 ) 


( 4 ) 


( 5 ) 
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Material & 
Equipment 












Other 
(Specify) 












Client Time & 
Other Client 
Inputs 













TOTAL t $ 



Source: Adapted from Levin, 1975? p. 101. 

Figure 1. Illustrative Framework for Cost Accounting in 
Educational Programs. 



Appropriate Costs of Evaluation 

Clearly, evaluations carry both direct and indirect costs, 
and both can be calculated. But iust what costs are 
"appropriate" (i.e., what level of resources should be devoted to 
asking and answering questions about how well programs work or 
what they achieve)? This question can be approached in two 
ways. At a macro level, one can compare the cost of evaluation 
with the cost of other program elements. That is, one can ask 
what the total costs of an evaluation should be relative to the 
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entity being evaluated. At a micro level, one can look at the 
way in which direct costs are allocated among the various budget 
categories and at the nature of the indirect costs associated 
with the conduct of the evaluation. Here we might ask what 
particular sorts of costs are worth incurring, given the nature 
of our evaluative interest. 

As noted in The Costs of Evaluation (Alkin & Solmon, 1983), 
the literature provides very few guidelines with respect to the 
macro level. Rusnell (1979) indicates that the evaluation cost 
should amount to 10 percent of program cost, a figure that had 
been recommended during the early years of the federal Title VII 
programs under the Elementary and Secondary Education Act 
(ESEA). More specifically, in the operation of these 
bilingual/bicultural programs during the early 1970s, it was 
suggested that funds for evaluation activities^hould constitute 
8 percent and funds for audit activities should constitute 2 
percent of programs costs. More recently, Drezek and his 
associates (1982) surveyed 55 LEAs and 14 SEAs and reported the 
proportions of funds allocated for evaluation in the various 
program proposals of those agencies. As expected, the 
percentages varied substantially from one program to another; the 
median low was 1.5 percent and the median high was 5.5 percent. 
In addition, respondents to the survey recommended a 4-8 percent 
range as desireable. 

Guidelines at the micro level are even more scarce. In a 
study designed to gather data on the direct costs of the typical 
evaluation, Alkin and Stecher (1983) asked a nationwide group of 
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evaluators to consider the costs associated with two types of 

evaluation (process/implementation and outcome/summative ) and 

with three budget levels ($25,000, $10,000, and $4,000). Further 

constraints were imposed on the hypothetical .examples, to insure 

that the cost estimates would be comparable. Overall (that is, 

for both types of evaluation and at all three budget levels), the 

average shares recommended for each category were these: 

Professional staff 70% 
Clerical/secretarial staff 16% 
External consultants 2% 
Materials, supplies, telephone 5% 
Data processing 3% 
Facilities 0% 
Travel 3% 

The specific recommended cost breakdowns varied widely, 
depending in part on evaluation type and on total budget level. 
Many of these differences are easily explained. For example, the 
percentage of direct costs allocated for travel was four to five 
times higher in implementation/process evaluations than in 
outcome/summative evaluations. Process evaluations typically 
require more site visits and more consultation with people in the 
field than do summative evaluations. 

Total budget level made a difference with respect to three 
of the seven cost categories. The first was data processings 
"Data processing expenditures rose dramatically as the total 
evaluation budget increased from $4,000 to $10,000 and continued 
to rise in dollars (but not as a percentage of the total budget) 
when the total budget increased to $25,000" (Alkin & Stecher » 
1983, p. 7). One would expect data-processing costs to be low in 
a total evaluation budget of $4,000, since most of the analytic 




work connected with such a low-cost evaluation would probably be 
done on a hand calculator. Moreover, the percentage of the 
budget allocated to data processing would probably not increase 
beyond a certain level, due to economies of sc^ale: The initial 
outlay is high, but incremental costs for added units are 
relatively low. 

The second cost category where the proportionate allocation 
increased with the size of the total budget was travel: 0 
percent of the $4,000 budget, 2-3 percent of the $10,000 budget, 
and 5-8 percent of the $25,000 budget. Similarly, the proportion 
allocated for the third cost category — clerical and secretarial 
staff — rose as the size of the budget increased, from only 5 
percent at the $4,000 level to 20 percent at the $25,000 level. 
These systematic differences make clea| the extent to which 
indirect dollar costs are likely to be present in various 
evaluation budgets. The amount of data preparation, reporting, 
and just plain bureaucratic red tape seems to grow as the scope 
of the evaluation grows. Thus, a greater proportion of 
secretarial time is required at higher budget levels. Another 
potential explanation — and one that may be more relevant — is 
that, at small budget (or direct cost) levels, secretaries and 
clerical personnel are not easily divisible into arbitrary 
smaller budgeting units. Thus, these services are often provided 
by other projects or even by the school district itself. As a 
result, the secretarial costs of small evaluation projects tend 
to be disregarded when costs of evaluations are contemplated. 
They actually represent an indirect cost which may, on the basis 
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of our findings, be fairly substantial in small-scale 
evaluations. 

Our research on the costs of evaluation has convinced us 
that it is difficult, if not impossible, to. generalize about the 
direct or indirect costs associated with evaluations of different 
types. Idiosyncratic features affect the costs of evaluation and 
the extent of direct and indirect costs. These features can be 
grouped -into four categories which, though occasionally 
overlapping, offer a convenient schema for purposes of 
discussions context, administrative organization, support 
services and facilities, and administrative expectations and 
predispositions. 

One example that immediately comes to mind with respect to 
context is choosing between an internal and an external 
evaluator. The selection of an internal evaluator may impose 
indirect cost burdens on other units within the organization. 
Conversely, if an external evaluator is chosen, many more of the 
costs will be direct because of the necessity for a contract. 

* 

The administrative organization of the school district may 
impose different constraints upon the evaluation and thus affect 
its cost. If a complicated variety of approvals (for 
instruments, testing schedules, and so forth) and other 
administrative procedures are required, not only will the 
evaluator have to spend more time on the evaluation (a direct 
cost) but also other people in the organization will have to 



spend more of their time processing requests and making approvals 
(an indirect cost)* 

Likewise, the nature of the support services and facilities 
available within a school district may have cost implications. 
For instance, the availability of computer services converts what 
would otherwise be a direct cost into an indirect cost. 
Similarly, if the evaluator has access to unused < r underutilized 
facilities within the district, then the direct cost of renting 
office space will be saved. \ 

Finally, the expectations of program administrators — and 
especially their predisposition to evaluation use — can affect 
both the direct and the indirect costs of an evaluation. 
Clearly, if the administrators who are the potential users of 
evaluation findings are hostile toward the evaluation and 
inclined to dismiss its findings, the evaluator will have to 
spend more time and energy trying to modify these 
predispositions, thus increasing both direct and indirect costs. 
(This issue is discussed further in the next section.) 

Costs in Relation to Benefits 

Most authorities agree thnt the real issue is neither the 
absolute dollar cost of an evaluation nor the cost of the 
evaluation relative to the cost of the program being evaluated. 
The real issue is the relationship between the costs incj|rred by 
and the benefits that accrue from the evaluation. For example, 
Scriven (1974) notes that evaluations should be "cost free," 
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implying that they should at least pay for themselves in the 
benefits they provide. Clearly, Scriven is advocating that costs 
and benefits should be compared by subtracting costs from 
benefits (B-C). Similarly, in Standards f or * Evaluations of 
Educational Programs , Projects , and Materials (1981), the Joint 
Committee on Standards for Educational Evaluation says that "the 
evaluation should produce information of sufficient value to 
justify the resources expended " (p. 60 j emphasis added). * The 
Joint Committee's guidelines further indicate that one should 
thoroughly investigate the costs and benefits of an evaluation 
before deciding to undertake it, adding that the evaluator should 
"conduct evaluations as economically as possible" (p. 61). 

' If one accepts this point of view, a critical question 
becomes: How can the benefits deriving from an evaluation be 
determined? In other words, what constitutes an appropriate 
measure of benefits? Both Scriven and the Joint Committee would 
probably say that the dollar savings produced by the evaluation 
is the most appropriate measure. If the evaluation has resulted 
in recommendations as to how the program can be conducted in a 
less costly manner (with no reduction in the quantity or quality 
of educational outputs), and if the cost savings exceed the costs 
of the evaluation, then the cost of the evaluation is justified. 

This simple notion has been successfully practiced by^ Steven 
Frankel, Director of the Montgomery County (Maryland) Department 
of Education Accountability. In essence, by selecting for 
evaluation only those projects and services which are readily 
amenable to the demonstration of cost savings (primarily business 
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service activities), his Department has dramatically increased 
its total budget for evaluation and has freed up resources for 

conducting evaluations in which cost savings are not involved. 

4 •> 

The simple dollar-savings criterion is. obviously inadequate 
in those instances where cost economies cannot be demonstrated 
but where the evaluation still confers benefits, though of a less 
monetary nature. For example, an evaluation may result in 
recommendations on ways to increase instructional effectiveness 
which, if implemented, will lead to an increase in student 
achievement. Such an evaluation may well be considered vorth the 
dollar cost. Several issues arise here. The first is cost 
effectiveness: the dollar cost of the evaluation relative to the 
resulting increase in instructional effectiveness (and hence in 
student achievement). The second issue relates to the 

« 

economist's concern with indirect costs in the second sense of 
the term: opportunity costs. Granted that the benefits of an 
evaluation — in terms of increased student achievement — exceed 
the costs of the evaluation, it is nonetheless possible that 
alternative uses of the evaluation funds might have led to even 
greater benefits (i.e., higher student achievement). 

The third issue is more complicated: The recommendations of 
an evaluation are not always implemented, so one cannot always 
determine whether they produce benefits. Some authorities would 
take the position that, in such instances, one should consider 
the potential benefits that would have accrued, had the 
recommendations been implemented, and use those as a basis for 
judging the appropriateness of the costs. Other authorities 
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would maintain that, if the evaluation was not persuasive enough 
to convince decision-makers that its recommendations should be 
implemented, then the evaluation has failed and cannot be 
regarded as having produced benefits. The question comes down to 

N 

this: Should one look at potential or at actual benefits? Is 
evaluation only a type of research, whose benefits are to be 
judged by the nature of the research findings? Or is evaluation 
a decision-oriented interactive process, whose benefits are to be 
judged not so much by the recommendations it makes but by the 
extent to which it beneficially informs the decision process? 
Obviously, the answers one gives to these questions will 
determine, in large part, just how one calculates the 
cost-benefit equation. 

Cost Implications of High-Utilization Factors 

The Evaluation Productivity Project has, over a period of 
years, been concerned with identifying those factors associated 
with instances of high evaluation utilization (see, for example, 
Alkin, Daillak, & White, 1979; Daillak, 1980; Stecher, Alkin, 6 
Flesher, 1981). During the current fiscal year, we have 
completed a handbook for evaluation decision-makers that includes 
a factor framework (Alkin, Jacobson, Burry, Ruskus, White, & 
Kent, in press). 

One conclusion to emerge from our years of research is that 
the way in which the evaluation is conducted affects the extent 
to which its findings are utilized. As mentioned in the previous 
section, the real issue in judging the appropriateness of an 
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evaluation's cost is the extent to which the benefits exceed the 
cost. If the findings of an evaluation are not utilized at any 
level, then it is questionable whether even the most minimal 
evaluation costs are justified. At the same time, one must 
recognize that the very aspects of the evaluation most likely to 
increase its use are also likely to involve high costs in terms 
of both dollars and also the time and energy of almost everyone 
involved. 

The results of our studies have not been surprising. We 

0 

have found that the dedicated evaluator — the one who takes the 
time to understand! the political complexities of the evaluation 
situation, to consider the needs and interests of potential 
users, and to involve them in the planning and conduct of the 
evaluation — stands a better chance of seeing the findings of 
the evaluation utilized. But at the same time, this approach 
incurs some heavy costs, at least in terms of the evaluator's 
time and the time of program staff. Similarly, evaluation 
utilization increases when the potential user — the program 
administrator who commissioned the evaluation — is interested in 
the evaluation, is involved in the evaluation process, and is 
committed to integrating the evaluation findings into the 
decision process. Obviously, this degree of involvement on the 
part of potential users entails substantial indirect costs in the 
form of forgone opportunities to use their time in other, 
possibly more productive, ways. 

Let us look more closely at some of the factors just 
mentioned, starting with the political sensitivity of the 
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evaluator. The politically sensitive evaluator must be 
particularly attentive to the people within the organization, 
their place in the administrative structure, their special 
interests, and so forth. Such attentiveness may require a 
considerable outlay of the evaluator 's time in becoming familiar 
and dealing with the situation. Moreover, if sensitivity to 
multiple constituencies adds to what must be known, there may be . 
additional costs associated with extended data collection and 
data processing. In addition to these direct costs, there are 
indirect costs: for instance, the time which program . personnel 
spend interacting with the evaluator, the time which clients or 
pupils spend taking tests. 

The evaluator 's credibility is another factor that affects 
utilization. Credibility depends in part upon the evaluator 's 
credentials (e.g., academic or professional degrees, prestige or 
reputation, institutional affiliations, experience). Obviously, 
a highly credentialed evaluator costs more than a relatively 
uncredentialed evaluator: in consultant fees for external 
evaluators and in salaries for evaluators internal to the 
organization. Our research shows, however, that credibility is a 
function not only of the evaluator 's credentials at the outset of 
the evaluation but of the evaluator 's actions during the course 
of the evaluation: "As evaluators engage in their activities, 
they may come to be viewed as credible on a wider range of topics 
or credible to new audiences." (Alkin et al., 1979, p. 247). In 
short, an evaluator can build credibility, but this process is 
time-consuming and involves both direct and indirect costs. 



.As indicated earlier , the potential user is also a key 
figure in evaluation utilization. The more the potential user is 
concerned about the evaluation, involved in its conduct , and 
interested in its results, the greater the likelihood that the 
findings of the evaluation will be utilized. Obviously, this 
kind of user involvement has cost implications, especially in 
terms of indirect costs. -And the larger the number of potential 
users, the higher the costs. 

Not only evaluator and user characteristics but also certain 
project characteristics have cost implications. One example is 
the particular requirements that the organization imposes upon 
the evaluation. To the extent that these contractual obligations 
or written requirements are not directly relevant to the central 
concerns of the evaluation or the interests of the potential 
users, they may entail high costs — in terms of the time and 
research spent by the evaluator and the time spent by program 
personnel in interacting with the evaluator — and yet have np 
accompanying payoff in high utilization. 

The evaluation procedures, including design and methodology, 
obviously have cost implications. To the extent that these 
procedures are tailored to the particular needs of the program 
being evaluated and are applied rigorously, their costs may be 
high. These costs include outlays for materials, data 
processing, and so forth; the time and effort of the evaluation 
staff and program personnel; and client time spent in testing. 
Nonetheless, appropriate procedures are essential to higH 
utilization, insofar as the way in which the evaluation is 
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conducted influences potential users' perceptions of the 



^Finally, evaluation reporting, which is strongly related to 
utilization, has cost implications. To prepare a report that can 
be easily understood by all potential audiences and to 
disseminate it in a timely manner requires considerable effort on 
the part of the evaluator, and thus considerable cost. 
Similarly, follow-up procedures — designed to assure that users, 
understand the report and its recommendations — call for extra 
time and effort and therefore carry extra costs. Our research 
shows that simply preparing and distributing a report of the 
evaluation is not enough, if high utilization is desired. 
Bvaluators should regard evaluation as a process, not a product. 
They should view evaluation reporting as an almost-continuous set 
of activities designed to sensitize potential users to the 
information being developed, to prepare them for the findings 
that will emerge from the evaluation, and to encourage them to 
implement the recommendations. Without question, such' activities 
*il high costs, both direct and indirect. Nonetheless, they 
ssential to attaining high levels of evaluation use. 



These reflections on the costs of evaluation represent a 
brief summary of the work we have completed in this area and of 
the implications that can be drawn from it. In synthesizing our 
reflections, we have drawn heavily on The Costs of Evaluation 
(Alkin & Solmon, 1983). Many authors contributed chapters to 



evaluator 's credibility and of the quality of the evaluation. 



Summary 
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this volume, and some of their ideas are incorporated into this 
piece. Our perspectives on the costs associated with various 
utilization factors grew out of our previous effort in developing 
the utilization framework. Thus, the paper provides a synopsis 
of our work relative to the costs of evaluation, particularly 
evaluation which has a high potential for utilization. 
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